## 2 Literature Review
### Transformer-Based Method Application in Stock Price Prediction  
The Transformer architecture, due to its remarkable performance in the domain of natural language processing, has garnered widespread acclaim [13; 14]. In recent years, it has been incrementally applied to time series predictions, especially in financial contexts such as stock price forecasting [22]. In this section, the principles of time series prediction methods based on the Transformer will be elucidated, followed by a discussion of its application and advantages in the context of stock price prediction. Central to the Transformer is the self-attention mechanism, which allows the model to consider all positions in the input sequence when generating outputs, thus capturing long-range dependencies. For time series tasks such as stock price prediction, the ability to process these long-distance dependencies is pivotal, given that market shifts may be influenced by events from a distant past, as shown in Figure 3.  
Assuming there is a time series of stock price data \(X=(x_{1},x_{2},\ldots,x_{t})\), where \(x_{i}\) denotes the stock features on the \(i^{th}\) day, the aim is to predict the stock price on day \(t+1\)  
Figure 3: Application of Transformer used in series data.  
Within the Transformer, the self-attention mechanism initially computes a query vector \(q_{i}\), a key vector \(k_{i}\), and a value vector \(v_{i}\) for every position in this sequence:  
\[q_{i}=W_{q}x_{i} \tag{10}\]  
\[k_{i}=W_{k}x_{i} \tag{11}\]  
\[v_{i}=W_{v}x_{i} \tag{12}\]  
Here, \(W_{q}\), \(W_{k}\), and \(W_{v}\) are weight matrices to be learned. Subsequently, for the query vector \(q_{i}\) at each position, dot products are performed with all key vectors \(k_{j}\), which are then converted to weights via the softmax function:  
\[\alpha_{ij}=\text{softmax}\left(\frac{q_{i}\cdot k_{j}^{T}}{\sqrt{d_{k}}}\right) \tag{13}\]  
In this equation, \(d_{k}\) represents the dimension of the key vector. These weights, \(\alpha_{ij}\), dictate the attention allocated to the \(j^{th}\) day input when generating the output for the \(i^{th}\) day. Through these weights, a context vector \(c_{i}\), which is a weighted representation of the entire input sequence, is derived:  
\[c_{i}=\sum_{j}\alpha_{ij}v_{j} \tag{14}\]  
This context vector \(c_{i}\) provides a weighted representation concerning the entire input sequence, which is subsequently used for the prediction of the stock price on day \(t+1\). In stock price prediction applications, the self-attention structure of the Transformer offers several principal advantages. Primarily, it captures the long-distance dependencies inherent in stock price data, a feat challenging for traditional architectures such as RNNs or LSTMs. Additionally, unlike RNNs, which process sequentially, the Transformer can handle the entire sequence in parallel, vastly enhancing computational efficiency. Moreover, the multi-head attention mechanism permits the model to discern patterns in the data from multiple perspectives, thereby bolstering its representational power. In conclusion, Transformer-based time series prediction methods present a potent and efficient tool for stock price forecasting. Through the self-attention mechanism, it adeptly captures intricate patterns and long-distance relationships within stock price data, culminating in more accurate predictions.