## 3 Data and Method
### Proposed Method
#### 3.2.2 Memory Attention Module  
The Memory Attention module stands as a pivotal component within the Memory-Transformer model. This module was innovatively conceived to facilitate the model's capability to capture and retain long-term dependencies inherent in time series data. To achieve this end, a distinctive memory unit has been introduced. When integrated with the conventional self-attention mechanism, as shown in Figure 6, they collectively enable the capture of enduring trends.  
Diving deeper into the architecture, the memory unit is conceptualized as a matrix \(M\in\mathbb{R}^{d\times n}\), where \(d\) represents the dimensionality of each memory item and \(n\) indicates the total number of memory items the unit can store. Every memory item, denoted by \(m_{i}\in\mathbb{R}^{d}\), corresponds to data information at a specific time point \(t_{i}\). At each time step, an update to the memory unit is executed. Specifically, given the current input \(x_{t}\) and its associated hidden state \(h_{t}\) (originating from the Transformer's encoder), the memory unit's update can be expressed as:  
\[M_{\text{new}}=\text{Update}(M,h_{t},x_{t}) \tag{18}\]  
In the above, Update serves as a function tasked with selectively updating certain memory items in the memory unit based on the current input and hidden state. Such a design ensures that the model retains pivotal information beneficial for prediction while discarding inconsequential or redundant data. Moving forward, the process of leveraging  
Figure 6: Illustration of the Memory Attention module.  
this memory unit within the self-attention mechanism is elaborated. Contrasting with the traditional self-attention mechanism, the primary distinction of the Memory Attention mechanism is its consideration of the memory unit's information when calculating attention weights. Specifically, for every query vector \(q_{i}\), dot-product operations are conducted not just with all key vectors \(k_{j}\) of the current time step but also with all memory items within the memory unit. Hence, the formula for the attention weight computation is:  
\[\alpha_{ij}=\text{softmax}\left(\frac{q_{i}\cdot(k_{j}+M_{j})^{T}}{\sqrt{d_{k} }}\right) \tag{19}\]  
Here, \(M_{j}\) designates the \(j\)-th memory item within the memory unit. By adopting such an approach, the model, when determining attention weights, takes into account not only the current input information but also incorporates pivotal information from the past.  
The mathematical rationale behind this design is its empowerment of the model to effectively capture long-term dependencies. As the model processes data at a specific time point, it integrates both current information and crucial details from previous instances. This integration aids the model in comprehending the contextual backdrop of the current data, leading to enhanced predictive accuracy. Moreover, from an application perspective, this design showcases significant advantages in the domain of stock price prediction. In the realm of stock markets, current prices are often influenced by critical past events, which could be scattered across an extended timeline. By introducing a memory unit, the Memory Attention module aids the model in pinpointing these dispersed pivotal events, thereby offering a more comprehensive understanding of the current market scenario and yielding more accurate predictions.