## 3 Methodology
### Linear Transformer Transformation Process  
In a standard transformer, the self-attentive mechanism simultaneously operates on the entire input sequence [22], which can be computationally intensive for long lines. However, in the linear transformer, the input sequence is divided into smaller blocks or segments, and the attention mechanism is applied to each block separately. This approach can handle extended arrangements more quickly and efficiently without sacrificing the modeling capability of the attention mechanism [23]. In this paper, we propose a linearly optimized transformer structure while retaining the essential multiple attention mechanisms in the transformer structure to solve problems stemming from the lack of historical data and accuracy of prediction encountered in the present stage of financial time-series prediction [24]. The matrix operations involved in the encoder's multiple attention mechanism will make the model's time complexity squared. To reduce the complexity of the model and make it better adapted to time-series forecasting, this paper replaces the multiple attention module in the encoder with the linear Fourier transform. Its structure diagram is shown in Figure 3.  
Figure 2: Encoder schematic. The replacement Fourier transform equation is as follows. Let the implicit layer dimension be \(d_{m}\) and the sequence length be \(N\). In the case of a long sequence input, the self-attentive computation requires the storage space of the \(NxN\) attention distribution matrix. The self-attentive module will cause a bottleneck in the model performance [25]. The price series in this paper are discrete price data with hourly and minute frequencies in the time domain and are suitable for the discrete Fourier transform:  
\[F(n)=\sum_{i=0}^{N}f(t)\mathrm{e}^{-\mathrm{i}\frac{2\pi n}{N}t}, \tag{1}\]  
\[F(d_{s},d_{t})=\frac{1}{MN}\sum_{T=0}^{M}\left(\sum_{t=0}^{N}f(T,t)\mathrm{e}^ {-\mathrm{i}\frac{2\pi n}{N}t}\right)\mathrm{e}^{-\mathrm{i}\frac{2\pi t}{M}T}. \tag{2}\]  
where the input encoding vector dimension of the model is [Batch-size, \(d_{s},d_{t}\)]. The Fourier transform is applied to the sequence length dimension \(d\). It is equivalent to decomposing the sequence into multiple sinusoidal superposition forms in this dimension; then, the Fourier transform is performed on the feature variable dimension \(d_{t}\), which is again decomposed into sinusoidal superposition forms in the feature variable dimension. We use the two-dimensional Fourier transform to transform the time-domain sequence into the frequency domain and perform feature extraction based upon it. Compared with the multi-headed self-attentive mechanism, this paper adopts a linear transformer using the Fourier transform module. This linear structure, without introducing parameters, does not increase the complexity of the model but also reduces the model complexity to the linear level [26]. In this paper, the decoder part of the transformer is replaced and the replacement process is shown in Figure 4.  
Figure 3: Linear transformer structure flowchart. Except for the Fourier transform, the formulas for calculating the remaining parts of the linear transformer, which are the same as the standard transformer, are as follows. 1. Self attention: \[\text{Attention}(Q,K,V)=\text{softmax}\bigg{(}\frac{QK^{T}}{\sqrt{d_{k}}}\bigg{)}V,\] (3) where \(Q\) is the query matrix; \(K\) is the content to be attended; \(QK^{T}\) is the dot product operation, which calculates the attention weight for \(Q\) on \(V\); the purpose of scaling by \(\sqrt{d_{k}}\) is to avoid an excessively large dot product because the gradient after \(softmax\) will be slight when the dot product is excessively large. Another advantage of \(softmax\) is that it facilitates the back-propagation gradient calculation while smoothing the results to the 0-1 interval. The initial \(Q,K,V\) are the same, resulting from the sum of word embedding and position embedding.