# LTE: Lightweight Transformer Encoder for Orbit Prediction
## 2 Materials and Methods
### Prediction Model
Additionally, we also removed the LN. The decision to eliminate the LN was motivated by the characteristics of our dataset, which exhibits regular and repetitive patterns. Since our data are already normalized through a preprocessing step, LN may add unnecessary complexity and potentially over-regularize the model, limiting its ability to fully learn underlying patterns. By removing this component, we aimed to simplify the model and allow it to focus more effectively on learning the core patterns in the data. Furthermore, the absence of LN helps to improve the gradient flow, allowing the model to train faster and more freely without being overly constrained by normalization. As a result, the removal of LN led to a better performance, demonstrating that in the case of simple, pre-normalized datasets like ours, additional normalization layers may not always be necessary and can  
Figure 3: Comparison of the architectures of the Transformer encoder (**a**) and the LTE model (**b**). To improve prediction performance and reduce computational resources, the LTE model removes two unnecessary layers from the Transformer Encoder: PE and LN. sometimes hinder model optimization. The efficiency analysis of removing PE and LN is presented in Section 3.3. We now explain the process of the LTE in terms of its algorithms. The multivariate sequence input \(Y\in\mathbb{R}^{B\times N}\) becomes \(Q\in\mathbb{R}^{B\times N},K\in\mathbb{R}^{B\times N}\), and \(V\in\mathbb{R}^{B\times N}\) is computed in the form of Equation (3) [23]. Here, \(B\) means batch size and \(N\) means the dimension of the dataset. First, the dot product of \(Q\in\mathbb{R}^{B\times N}\) and \(K\in\mathbb{R}^{B\times N}\) is performed, and this value is divided by \(d_{k}\) to reduce the variance of the weights caused by the dot product. The explanation of the \(d_{k}\) value is addressed in Equation (4) in the next paragraph. The softmax function normalizes the computed \(Q\in\mathbb{R}^{B\times N},K\in\mathbb{R}^{B\times N}\) values so that their total sum becomes 1 and assigns relative importance to each value when computing with \(V\in\mathbb{R}^{B\times N}\). \[\text{Attention}(Q,K,V)=\text{softmax}\bigg{(}\frac{QK^{T}}{\sqrt{d_{k}}} \bigg{)}V \tag{3}\]  
The Transformer model uses Multi-Head Attention, a variant of Attention, which allows the model to simultaneously attend to different subspaces of information at various positions, enabling it to learn the data's information more comprehensively. The computation of Multi-Head Attention begins by dividing the input into multiple heads, each performing independent attention operations. The parameter of the number of heads (\(n_{\text{head}}\)) determines how many attention heads are used, allowing the model to distribute its focus across different parts of the input and learn diverse representations. Division into multiple heads enables the model to capture more detailed relationships within the data. To perform the operations of Multi-Head Attention, we first divide \(Q,K,V\) into \(Q_{i}\in\mathbb{R}^{B\times\frac{N}{n_{\text{head}}}},K_{i}\in\mathbb{R}^{B \times\frac{N}{n_{\text{head}}}}\), and \(V_{i}\in\mathbb{R}^{B\times\frac{N}{n_{\text{head}}}}\). Here, \(B\) means batch size, \(N\) means the dimension of the dataset, and \(n_{\text{head}}\) is how many heads are used to divide the dataset during Multi-Head Attention. We then apply Equation (4) [23]. We can now define \(d_{k}\) in Equation (3), which refers to the dimension of the data \(N\) divided by \(n_{\text{head}}\). \[\text{MultiHead}(Q,K,V) =\text{Concat}(\text{head}_{1},\text{head}_{2},\dots,\text{head} _{h})W^{O} \tag{4}\] \[\text{where}\quad\text{head}_{i} =\text{Attention}(QW_{i}^{Q},KW_{i}^{K},VW_{i}^{V})\]  
The values that pass through the Multi-Head Attention layer pass through a residual connection layer, followed by the Feed-Forward layer, and then through another residual connection layer to finally obtain the predicted result \(\Upsilon^{\prime}\in\mathbb{R}^{B\times N}\).