# LTE: Lightweight Transformer Encoder for Orbit Prediction
## 2 Materials and Methods
### Experiments  
For model training, we used approximately 4.8 million data points collected from the K3, K3A, and K5 satellites from January 2016 to December 2018. The dataset was divided by setting January to December 2016 as the training dataset, January to December 2017 as the validation dataset, and January to December 2018 as the test dataset. All the models in the experiments utilized the same training, validation, and test data. In Section 3.1, we present the results of comparing the prediction performance of our LTE model with that of various baseline models, along with comparisons of the execution time and the number of parameters used against those of Transformer-based models. The baseline models used in our experiments are listed below:  
* Support Vector Regressor (SVR) [28]: A machine learning model that uses support vector machines (SVMs) for regression tasks by finding the best-fitting hyperplane that minimizes the error within a predefined threshold.
* eXtreme Gradient Boosting Regressor (XGBR) [29]: An ensemble learning method based on gradient boosting, which builds a series of weak learners (decision trees) to improve predictive performance.
* Long Short-Term Memory (LSTM) [30]: A type of recurrent neural network (RNN) designed to model sequences and time series data by effectively learning long-term dependencies and preventing the vanishing gradient problem.
* Variational AutoEncoder (VAE) [31]: A generative model used for unsupervised learning that learns a probabilistic representation of the data by encoding them into a lower-dimensional space and then decoding them back into their original format.
* Gated Recurrent Unit (GRU) [32]: Another RNN variant similar to LSTM, but with a simpler architecture, using fewer gates to capture dependencies in sequential data while also addressing the vanishing gradient problem.
* Bi-directional Long Short-Term Memory (Bi-LSTM) [33]: An extension of LSTM that processes input sequences in both forward and backward directions, improving its ability to capture context from both past and future information in sequential data.
* Transformer [23]: A neural network architecture based on attention mechanisms that process sequential data without relying on recurrence, making it highly effective for tasks like machine translation and time series prediction.
* Transformer (encoder only): A model that separates the encoder structure from the Transformer architecture. The model first applies PE to the input, followed by Multi-Head Attention, residual connections, LN, and a Feed-Forward layer.
* Transformer (decoder only): A model that separates the decoder structure from the Transformer architecture. The model first applies PE to the input, followed by Multi-Head Attention, residual connections, and LN. Then, it passes through another Multi-Head Attention layer, residual connections, and LN, before passing through a Feed-Forward layer.  
We can categorize these 9 models into three groups: (1) machine learning-based models (SVR and XGBR), (2) conventional neural networks (LSTM, VAE, GRU, and Bi-LSTM), and (3) Transformer-based models (Transformer, Transformer (encoder only), and Transformer (decoder only)). Our experiments were conducted in different configurations within each group using various hyperparameters. Also, the experiments were conducted five times, and the average of the five results was recorded in the result table.  
All models, except for the machine learning-based models, used the Adam optimizer with a learning rate of 0.001, and the activation function was set to a Rectified Linear Unit (ReLU). Additionally, the batch size was set to 60, the number of hidden units to 64, and the dropout rate to 0.1. In the Transformer-based models, the \(n_{\text{head}}\) for Multi-Head Attention was set to 3 and 6, and the number of Transformer encoder and decoder layers was set to 1.