## 3 Data and Method
### Proposed Method
#### 3.2.1 Memory-Transformer  
With the rapid advancements in deep learning techniques, the Transformer model has demonstrated superior performance across various domains, particularly in the realm of natural language processing. However, for time series data, especially in the context of stock price prediction, relying solely on the Transformer might be insufficient. In response to this challenge, this research introduces the Memory-Transformer model. Distinct from the conventional Transformer, the pivotal innovation of the Memory-Transformer is the incorporation of a memory unit. The primary intention behind this memory unit is to facilitate the model's capability to retain and recall historical data over extended periods. In the financial market, prolonged historical data often encapsulate invaluable insights, which are crucial for predicting future stock trends, as illustrated in Figure 5.  
Figure 4: Illustration of the whole method proposed in this paper.  
Mathematically, we utilize a time series of stock data \(X=(x_{1},x_{2},\ldots,x_{t})\), where \(x_{i}\) denotes the stock features of the \(i\)-th day. In the standard Transformer model, for every time point, a query vector \(q_{i}\), a key vector \(k_{i}\), and a value vector \(v_{i}\) are computed. In contrast, the Memory-Transformer introduces an additional memory vector \(m_{i}\) derived from the memory unit:  
\[m_{i}=f(M,x_{i}) \tag{16}\]  
Here, \(M\) represents the entire memory unit and \(f\) is a function designed to extract memory associated with the current time point \(x_{i}\). When computing self-attention weights, the Memory-Transformer considers not only the query, key, and value information from the current input but also integrates the memory vector \(m_{i}\). Specifically, the attention weight computation is given by:  
\[\alpha_{ij}=\text{softmax}\left(\frac{q_{i}\cdot(k_{j}+m_{j})^{T}}{\sqrt{q_{k} }}\right) \tag{17}\]  
Noticeably, the key vector \(k_{j}\) is augmented here, amalgamated with the memory vector \(m_{j}\). This ensures that when determining attention weights, the model not only contemplates the current input but also prior pivotal information. This design is profoundly significant. In the stock market, current prices are often influenced by pivotal past events. Such events might not only pertain to recent occurrences but could date back significantly. By integrating the memory unit, the Memory-Transformer captures these long-standing historical trends, leading to more accurate predictions. Furthermore, introducing the memory unit not only amplifies the model's long-term memory capacity but also expedites its convergence rate. The model no longer necessitates learning all historical information from scratch, instead leveraging information already retained in the memory unit.  
For tasks such as stock price prediction, the Memory-Transformer exhibits distinct advantages over the standard Transformer. Primarily, it excels at capturing long-term historical trends, an aspect where conventional RNNs or LSTM structures might falter. Additionally, by processing the entire sequence in parallel, it retains the computational efficiency intrinsic to the Transformer. Lastly, in conjunction with the memory unit and long-distance loss function, it places enhanced focus on long-term trends during training, aiming for precise predictions. In summary, the Memory-Transformer, with its memory unit incorporation, not only amplifies the model's long-term memory prowess but also upholds the Transformer's computational efficiency and representational strengths, positioning it as an ideal choice for stock price prediction tasks.  
Figure 5: Illustration of the Memory-Transformer architecture.