# LTE: Lightweight Transformer Encoder for Orbit Prediction
## 3 Results
### Efficiency Analysis: Removing Layer Normalization and Positional Encoding  
To verify the impact of removing PE and LN on the prediction performance of our model, we conducted an efficiency analysis on the presence or absence of PE and LN. Figure 5 is a graph comparing the values (0 to 400 data points) predicted by our model, LTE-h3, which has no PE or LN, with the ground truth values. The blue lines present the predicted values, and the yellow lines show the ground truth values. As shown in Figure 5, it can be confirmed that there is almost no difference between the actual data and the predicted data. On the other hand, Figure 6 is a graph comparing the values (0 to 400 data points) predicted by the LTE-h3 + PE model, which includes PE but no LN, with the ground truth values. As depicted in this figure, a distinct difference (prediction error) between the ground truth values and the predicted values can be observed. Similarly, Figure 7 is a graph comparing the values (0 to 400 data points) predicted by the LTE-h3 + LN model, which includes LN but no PE, with the ground truth values. As in Figure 6, a clear difference (prediction error) between the ground truth values and the predicted values can be observed. These graphs visually show that removing PE and LN leads to better performance. Figure 5: Plot comparing predicted values (0–400 data points) from LTE-h3, without PE and LN, with the ground truth. The blue line represents the predicted values and the yellow line represents the ground truth values. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Model** & **Execute Time (s)** & **Num. of Parameters** \\ \hline
**Transformer (decoder only)** & **1731** & 1174 \\ \hline
**LTE, ours** & \(\surd\)**1093** & \(\surd\)**1006** \\ \hline Improvement (Reduction Rate) & 36.86\% & 2.33\% \\ \hline \hline \end{tabular}
\end{table} 
Table 3: _Cont._Table 4 provides the exact numerical performance of various configurations of the LTE model, including different combinations of PE and LN, with different numbers of attention  
Figure 6: Plot comparing predicted values (0–400 data points) from LTE-h3 + PE, with PE and without LN, with the ground truth. The blue line represents the predicted values and the yellow line represents the ground truth values. Figure 7: Plot comparing predicted values (0–400 data points) from LTE-h3 + LN, without PE and with LN, with the ground truth. The blue line represents the predicted values and the yellow line represents the ground truth values. heads (\(n_{\text{head}}\)). The best-performing model is indicated in bold with a checkmark, while the second-best model is indicated in bold without a checkmark. The results clearly show that the best performance is achieved by our model, LTE-h3, without either PE or LN, when the number of heads is three. This configuration produces a prediction error of 0.0000238, demonstrating the superior performance of this simpler model. The second-best model was also our model, LTE-h6, without either PE or LN, when the number of heads is six, resulting in a prediction error of 0.0000288. Excluding the LTE model, the best-performing model is the Transformer encoder, with both PE and LN included, which achieved a prediction error of 0.0000299 with six heads (LTE-h6 + PE + LN). The next-best-performing model is also the Transformer encoder with both PE and LN, but with three heads (LTE-h3 + PE + LN), which achieved a prediction error of 0.0000499. This result suggests that while the Transformer encoder with both modules performs reasonably well, it cannot outperform our LTE model. On the other hand, the performance of models using either PE or LN alone is notably worse. For instance, with PE alone, the prediction error was 0.1211978 for the LTE-h3 + PE model and 0.0836508 for the LTE-h6 + PE model. Similarly, using LN alone resulted in prediction errors of 0.0737088 for LTE-h3 + LN and 0.0712439 for LTE-h6 + LN. These results indicate that when complex data are introduced through PE, it is necessary to normalize them with LN to maintain the model's performance.