## 3 Construction and Training of the Multivariate Attention Neural Network
### Structure of the Neural Network  
In recent years, due to the rise of Chat-GPT, transformers (including improved versions such as iTransformer, Informer, Autoformer, Crossformer, etc.) have received significant attention in the fields of natural language processing (NLP) and computer vision (CV). In transformer-based architectures, the most crucial portion is the multi-head attention mechanism. The multi-head attention mechanism captures the dependencies between different variables by applying different weights. In recent years, some scholars have found that in NLP and CV contexts, network architectures based on a multi-head attention mechanisms exhibit better capacity than conventional neural networks such as RNN, DNN, and CNN. Inspired by the relevant research work, in this paper we construct a neural network model based on a multi-head attention mechanism, which is applied to solve the orbit deviation propagation problem in the Earth-Jupiter transfer process. The structure of the Multivariate Attention-based Neural Network (MANN) constructed in this paper is shown in Figure 4.  
Figure 3: Optimized Earth–Jupiter transfer orbits. In Figure 4, the MANN developed in this paper is similar to the encoder module of the standard Transformer. As shown on the left side of Figure 4, the MANN consists of an embedding module, D (Depth) processing modules e (shown by gray in Figure 4) connected sequentially, and a projection module. Each processing module contains a multivariate attention layer, two normalization layers, and a feed-forward layer. The main role of the embedding module is to expand the sample input \(\mathbf{s}^{50}\in\mathbb{R}^{50}\) into a dim-dimensional vector \(\mathbf{s}^{embedding}\in\mathbb{R}^{dim}\), with the expression provided as follows:  
\[\mathbf{s}^{embedding}=\text{Embedding}(\mathbf{s}) \tag{25}\]  
where \(dim\) is a user-defined parameter. At the multi-head attention mechanism layer, for the i-th attention mechanism, the calculation process is as follows:  
Step 1: Calculate the three vectors \(\mathbf{Q}_{i},\mathbf{K}_{i}\), and \(\mathbf{V}_{i}\) using linear transformation:  
\[\mathbf{Q}_{i}=\mathbf{W}_{Q}^{i}\mathbf{s}^{embedding} \tag{26}\]  
\[\mathbf{K}_{i}=\mathbf{W}_{K}^{i}\mathbf{s}^{embedding} \tag{27}\]  
\[\mathbf{V}_{i}=\mathbf{W}_{V}^{i}\mathbf{s}^{embedding} \tag{28}\]  
where \(\mathbf{Q}_{i},\mathbf{K}_{i},\mathbf{V}_{i}\in\mathbb{R}^{dim}\) and \(\mathbf{W}_{Q}^{i},\mathbf{W}_{K}^{i},\mathbf{W}_{V}^{i}\in\mathbb{R}^{dim\times dim}\). Step 2: Calculate the scores by the sum dot product of \(\mathbf{Q}_{i}\) and \(\mathbf{K}_{i}\):  
\[\mathbf{S}_{i}^{0}=\mathbf{Q}_{i}\mathbf{K}_{i}^{T}. \tag{29}\]  
Step 3: Scale the scores calculated in Step 2. In this paper, the scores are divided by 8, which allows the MANN to have a more stable gradient. Then, the result is normalized by softmax:  
\[\mathbf{S}_{i}^{1}=\text{Softmax}(\mathbf{S}_{i}^{0}/8)=\text{Softmax}(\mathbf{Q}_{i}\mathbf{K }_{i}^{T}/8). \tag{30}\]  
Step 4: Multiply the output from Step 3, \(\mathbf{S}_{i}^{1}\), with the vector \(\mathbf{V}_{i}\) to obtain the output of the i-th head:  
\[\mathbf{z}_{i}=\mathbf{S}_{i}^{1}\mathbf{V}_{i}=\text{Softmax}(\mathbf{Q}_{i}\mathbf{K}_{i}^{T}/8 )\mathbf{V}_{i}. \tag{31}\]  
Figure 4: Structure of the designed MANN.