# A Transformer-Based Channel Estimation Method for OTFS Systems
## 3 Proposed Transformer Estimation Algorithm  
### Architecture of the Proposed Channel Estimation Algorithm  
Our proposed Transformer estimation structure is shown in Figure 3. The Transformer estimation algorithm is divided into two phases: offline training and online prediction. During the offline training phase, the neural network is trained using a substantial amount of channel data. This training process aims to optimize the parameters of the Transformer neural network for accurate channel estimation. In the subsequent online prediction phase, the preliminary result obtained from the threshold channel estimation algorithm is utilized as input to the trained neural network. By feeding this preliminary result into the network, the algorithm produces a more accurate output for channel estimation. Figure 3: Transformer estimation structure. In [25], a transformer based on a self-attentive mechanism was first proposed. As shown in Figure 3, the Transformer estimation structure consists of two parts: the encoder and the decoder. To balance performance and computational complexity, our Transformer estimation model retains the encoder-decoder structure of the original model and changes the number of encoder and decoder layers from six layers to two layers. In the encoder, the time series data \([h_{t-L},h_{t-L+1},\cdots,h_{t-1}]\) are mapped into multiple high-dimensional vectors through the feedforward fully connected layer at the input, where \(L\) denotes the time step. Subsequently, the temporal position information of the series is encoded in the position encoding area. The role of position encoding is to provide position information to the model through the linear variation of sin and cos functions. The positional encoding can be represented as  
\[PE(pos,2i)=\sin(pos/10,000^{2i/d_{\text{model}}}) \tag{12}\]  
\[PE(pos,2i+1)=\cos(pos/10,000^{2i/d_{\text{model}}}) \tag{13}\]  
where \(pos\) represents the location index of information for each moment of the series, \(i\) represents the dimensional index of high-dimensional vectors obtained by mapping the time series through the input layer, and \(d_{\text{model}}\) is the dimension of high-dimensional vectors. Then, the positional encoding information is added to the output of the input layer. Concatenating these vectors together yields a matrix, \(H^{\prime}=[h^{\prime}_{t-L+1},h^{\prime}_{t-L},\cdots,h^{\prime}_{t}]\), as the input, and three vectors are generated from \(H^{\prime}\) at each moment, which are the query vector, \(q\), key vector, \(k\), and value vector, \(v\)[26]:  
\[ \begin{array}{l}q_{t}=W^{q}h_{t}\\ k_{t}=W^{k}h_{t}\\ v_{t}=W^{n}h_{t}\end{array}  \tag{14}\]  
where \(W\) denotes the weighting matrix. The vectors at each moment are spliced into matrices: \(Q,K,V\). Matrices \(Q,K,V\) are calculated by the self-attentive mechanism to obtain the estimation results:  
\[Attention(Q,K,V)=softmax(\frac{QK^{T}}{\sqrt{d_{k}}})V \tag{15}\]  
The multi-headed attention mechanism is the integration of multiple independent attention modules, which is similar to multiple convolutional kernels in convolutional neural networks (CNN) to help the network extract richer features:  
\[MultiHead(Q,K,V)=Concat(head_{1},\cdots,head_{i})W_{o} \tag{16}\]  
where \(Concat(\bullet)\) denotes the matrix splicing operation, and \(W_{o}\) denotes the weighting matrix. Finally, the residual network is applied to solve the problems of gradient disappearance and weight matrix degradation. The results obtained from the multi-attention mechanism are subjected to residual connection layer and layer normalization operations to obtain the final outputs. In [27], ResNet was proposed to solve the problem of the difficult optimization of multilayer neural networks. The input-output relationship of the ResNet can be written as  
\[G(x)=F(x)+x \tag{17}\]  
where \(x\) denotes the input, and \(F(x)\) represents the nonlinear variation function.