# LTE: Lightweight Transformer Encoder for Orbit Prediction
###### Abstract
They generated a physics model that predicts future data points using historical data points. The prediction errors obtained from the physics model were fed into the GBRT model along with the points' actual values. Here, the GBRT model estimates the errors predicted by the physics model. As a result, the physics model improves its performance by adjusting its predictions using the prediction error estimated by the GBRT model. They demonstrated that their model outperformed Artificial Neural Network (ANN), and Gaussian Process (GP) models. The research carried out by [PERSON] and [PERSON], which uses only a machine learning model, and the research performed by [PERSON], which combines both a physics model and a machine learning model, each improved the performance of a prediction model using historical data. However, this research is closer to traditional numerical optimization and statistical techniques. Thus, it cannot effectively handle large datasets or complex patterns. Other research has explored recurrent neural network (RNN)-based models to predict satellite orbits and spatial targets [15]. Moreover, several studies have applied Long Short-Term Memory (LSTM) models, significantly reducing orbit prediction errors [16; 17; 18; 19; 20; 21; 22]. For example, [PERSON] and [PERSON] [15] predicted the orbit of spatial targets using an LSTM-based model combined with the Particle Swarm Optimization (PSO) algorithm. The PSO algorithm simulates the movement of particles in the search space to find the optimal solution within that space. The PSO-LSTM model calculates the current fitness value of space objects during training based on the mean absolute error, and it determines the position and velocity of space objects. Through this approach, the authors demonstrated that the PSO-LSTM model outperforms the traditional LSTM model's prediction performance. [PERSON] et al. [19] aimed to predict satellite orbits using TLE set conversions based on LSTM. They preprocessed historical data from the TLE set to obtain velocity and position vectors, aiming to predict the satellite's orbit using these two values. Then, they used the velocity vector as input to the LSTM model to predict the position vector and vice versa. The authors demonstrated that the proposed approach predicted both position and velocity vectors with a high accuracy of 98%. However, both studies utilized LSTM to predict the orbits of spatial targets or the orbit of satellites but did not consider factors like execution time or model parameter counts. More recently, growing attention has been directed toward orbit prediction using the Transformer model, which has demonstrated remarkable performance in Natural Language Processing (NLP) as well as time series forecasting. Introduced in 2017, the Transformermodel [23] is useful in parallel processing as it performs independent calculations across all elements in a sequence. Its attention mechanism captures the interactions between all positions in the input sequence, providing a more comprehensive context compared to earlier time series models. In this work, we leverage Transformer Networks for orbit prediction, as these have demonstrated promising results in time series prediction. We propose the Lightweight Transformer Encoder (LTE) model, which predicts satellite orbits with high performance while reducing execution time and model parameters. Our main contributions are as follows: We introduce the LTE model, specifically adapted for satellite orbit prediction by modifying the Transformer Encoder structure. The modifications include the removal of Positional Encoding (PE) and Layer Normalization (LN) to prevent the distortion of input data and over-normalization. Additionally, using real-world data from the Korea Multi-Purpose Satellite (KOMPSAT) series, the LTE model showed significant improvements, reducing the Mean Squared Error (MSE) by up to 50.61% over comparable models. It also showed efficiency gains, with reduced execution time and model parameters.