# LTE: Lightweight Transformer Encoder for Orbit Prediction
## 2 Materials and Methods
### Prediction Model  
In Figure 2, each element of the dataset exhibits a unique period and wave shape, and a repetitive pattern is observed within each element individually. This suggests that an original Transformer with a complex structure, containing numerous parameters and layers, may not be necessary for orbit prediction. Therefore, we leverage only the encoder of the Transformer for our LTE model, excluding the decoder, and further removing unnecessary layers to better fit the characteristics of the dataset. Figure 3a represents the structure of the Transformer encoder, while Figure 3b illustrates the structure of the LTE model. The Transformer encoder applies PE to the input \(Y\) and passes it through the Multi-Head Attention layer. The output \(Z\) from the Multi-Head Attention layer is fed through a residual connection and the LN layer to obtain \(Z^{\prime}\). Subsequently, it passes through a Feed-Forward layer and once again through a residual connection and the LN layer to obtain the final output \(Y^{\prime}\). Unlike the Transformer Encoder, the LTE model passes the input \(Y\) through the Multi-Head Attention layer without applying PE. The value \(Z\) that passes through the Multi-Head Attention layer is added to the input \(Y\) in the residual connection stage, resulting in \(Z^{\prime}\). In this case, the Transformer encoder applies both residual connection and LN, but the LTE model does not apply LN. Next, \(Z^{\prime}\) passes through the Feed-Forward layer and then through another residual connection stage to obtain the final output \(Y^{\prime}\). Even in this case, the LTE model applies only residual connection, without applying LN. The final output, \(Y^{\prime}\), represents the prediction of the 6 elements included in the input \(Y\) at a given time. PE is a method used in the original Transformer model to encode the position of each token in a sequence, since Transformers process input data in parallel and do not inherently track order like RNNs. By adding positional information to each token's embedding, using sine and cosine functions at different frequencies, the model can capture both short-term and long-term dependencies in the sequence. This allows the Transformer to understand the order of tokens and properly model relationships across time or sequence positions. However, in cases where the data exhibit repetitive patterns or do not require complex positional relationships, PE can unnecessarily complicate the model by introducing artificial patterns and higher frequencies, which can lead to overfitting or poor generalization. Therefore, careful consideration should be given to whether PE is truly beneficial for a given task. In Figure 4, we can observe the effect of applying PE to the Semi-major axis data. The yellow line, representing the original data without PE, shows a smooth and regular pattern with a relatively low frequency, which aligns with the natural fluctuation expected from satellite orbit dynamics. However, when PE is applied, as shown by the blue line, the data become significantly more complex, introducing higher-frequency components and irregular patterns. This added complexity distorts the original data, altering the natural characteristics of the Semi-major axis. Such distortion can mislead the model during training, causing it to focus on artificial frequencies rather than true orbital patterns. This not only complicates the learning process but also risks overfitting, where the model performs well on the distorted training data but fails to generalize to new, unseen data. Consequently, applying PE in our dataset may degrade the model's performance, making it harder to accurately forecast satellite orbits. Therefore, the use of PE for this type of data may not be appropriate, as it introduces unnecessary complexity and reduces the overall effectiveness of the model. Therefore, we removed PE from our model. This allows the model to focus on the inherent characteristics of the orbital data, improving both its learning efficiency and generalization to unseen data.