## 2 Basic Theory  
### Rwkv  
RWKV is a large-scale model with a novel neural network architecture that combines the characteristics of Recurrent Neural Networks (RNNs) [11] and Transformers [12]. By introducing a weighted key-value pair mechanism [13], it effectively captures contextual information from time-series data, empowering the model to discern and model interdependencies across distant time steps more effectively, while maintaining efficiency during feature extraction. Additionally, RWKV adjusts the memory length through a weighted mechanism, prioritizing the retention of important information relevant to the current time step, thereby enabling more targeted feature extraction. The flow of information within the RWKV architecture facilitates the seamless and efficient transmission of data across time steps, enabling the extraction of more nuanced and sophisticated textual features.  
The RWKV model is constructed based on stacked residual blocks, as shown in Figure 1a. Each residual block consists of a time-mixing subblock and a channel-mixing subblock. This design enables the simultaneous modeling of dependencies in both the feature and time dimensions, demonstrating the strengths of a recurrent structure. It allows the model to fully leverage historical information to enhance feature extraction at the current time step. The model incorporates a unique attention mechanism, which introduces a dynamic update mechanism, particularly the time-dependent operation. This significantly enhances numerical stability and alleviates the problem of vanishing gradients. Through this mechanism, the model is able to control the interaction of information between different time steps with finer granularity, thereby more precisely capturing key features. Additionally, the model structure integrates layer normalization techniques [14], which are crucial for stabilizing gradient flow and effectively addressing common issues such as gradient vanishing and gradient explosion during deep network training. This stability not only optimizes the model's training process but also further supports the stacking of deep networks, enabling the model to capture abstract features at different levels and extract more complex feature information.  
The core architecture of the RWKV model consists of four fundamental components, as illustrated in Figure 1b. The inherent components of the time-mixing and channel-mixing modules are as follows:  
* R (Receptance Vector): This component stores information from previous time steps. For example, if we are processing the sequence of words \"The cat sat on the mat,\" the receptance vector captures the context from earlier words (e.g., \"The cat\") to inform the understanding of the current word (\"sat\").
* W (Weight): This is a trainable parameter that adjusts the influence of past information on the current time step. In the example, the weight determines how much influence the previous word \"The\" should have when processing the current word \"cat.\"
* K (Key): The key helps to retrieve the relevant information from the past, providing a form of selective memory. For instance, when processing the word \"sat,\" the key mechanism would identify the relevant context from the earlier word \"cat.\"V (Value): The value represents the actual content or features associated with the key. In the example, when processing \"sat,\" the value would contain features or embeddings related to the word \"cat,\" such as its semantic meaning.  
These components work together to dynamically capture temporal dependencies and enhance the ability of the RWKV model to integrate both short- and long-term contextual information.