## 3 Model  
### CLIP Pre-Training Fusion  
CLIP (Contrastive Language-Image Pretraining) is a pretraining model developed by OpenAI, aiming to build a shared semantic feature space by learning the contrastive relationship between images and text. The design of CLIP is based on the idea that images and text should be treated as equally important inputs, and their semantic relationship can be established via joint training. Traditional computer vision models typically convert images into vector representations for tasks like image classification and detection. However, this approach fails to fully utilize the rich information within images and the interplay between images and text. CLIP innovatively treats images and text on equal footing, training the model to understand their semantic relationship using joint learning.  
Figure 7 shows the model architecture of CLIP used in this paper for pre-training. The CLIP model consists of two encoders, the text encoder and the graph encoder, which are responsible for processing text and image data, respectively. The text encoder utilizes a transformer model to transform the input text data into a high-dimensional text feature vector (\(T_{N}\)) representation. Our graph encoder employs a GNN model to learn and represent the syntactic information within the input graphs and convert it into a high-dimensional graph feature vector (\(l_{N}\)) representation.  
Next, we use a fully connected layer as the mapping function to map the text feature vector (\(T_{N}\)) and the graph feature vector (\(I_{N}\)) to a shared multimodal feature space. We aim to map them to a feature space of dimension D.  
Figure 7: CLIP model architecture.  
First, we apply a linear transformation to the text feature vector (\(T_{N}\)) to map it to a feature space of dimension D. This can be achieved using a weight matrix (\(W_{t}\)) and a bias vector (\(b_{t}\)), represented as:  
\[Z_{t}=W_{t}\times T_{N}+b_{t} \tag{6}\]  
Among them, \(Z_{t}\) is the intermediate feature vector after linear transformation of the text feature vector.  
Next, we apply the ReLU non-linear activation function to the intermediate feature vector (\(Z_{t}\)) to introduce non-linear transformation. This can be expressed as:  
\[A_{t}=\text{ReLU}(Z_{t}) \tag{7}\]  
Among them, \(A_{t}\) is the text feature vector after passing the non-linear activation function.  
Similarly, we perform similar linear transformation and non-linear activation function operations on the graph feature vector (\(I_{N}\)) to obtain the intermediate feature vector of the graph:  
\[Z_{i}=W_{i}\times I_{N}+b_{i} \tag{8}\]  
\[A_{i}=\text{ReLU}(Z_{i}) \tag{9}\]  
Among them, \(Z_{i}\) is the intermediate feature vector after linear transformation of the graphic feature vector, and \(A_{i}\) is the graphic feature vector after going through the nonlinear activation function.  
Finally, we combine the non-linearly transformed text feature vector (\(A_{t}\)) and graph feature vector (\(A_{i}\)) to obtain the shared multimodal feature vector. This is achieved by a linear transformation using an additional weight matrix (\(W_{m}\)) and bias vector (\(b_{m}\)), represented as:  
\[M=W_{m}\times[A_{t}:B_{i}]+b_{m} \tag{10}\]  
Among them, \([A_{t}:B_{i}]\) means splicing the text feature vector and the graphic feature vector in the feature dimension to obtain a vector with a 2D dimension. \(M\) represents the final multimodal feature vector.  
In this way, the GNN model and the transformer model can mutually incorporate information from both modalities, thereby better capturing the correlations and semantics between text and graphics. Finally, we fine-tune the transformer model, which incorporates information from both modalities, for machine translation tasks. This approach leverages the correlation and semantic information between the text and graphics to improve the quality and accuracy of machine translation, bringing new ideas and methods to the development of machine translation technology.