Simple Yet Effective: Structure Guided Pre-trained Transformer for Multi-modal Knowledge Graph Reasoning

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-06 12:01:47

Abstract: Various information in different modalities in an intuitive way in multi-modal knowledge graphs (MKGs), which are utilized in different downstream tasks, like recommendation. However, most MKGs are still far from complete, which motivates the flourishing of MKG reasoning models. Recently, with the development of general artificial intelligence, pre-trained transformers have drawn increasing attention, especially in multi-modal scenarios. However, the research of multi-modal pre-trained transformers (MPT) for knowledge graph reasoning (KGR) is still at an early stage. As the biggest difference between MKG and other multi-modal data, the rich structural information underlying the MKG is still not fully utilized in previous MPT. Most of them only use the graph structure as a retrieval map for matching images and texts connected with the same entity, which hinders their reasoning performances. To this end, the graph Structure Guided Multi-modal Pre-trained Transformer is proposed for knowledge graph reasoning (SGMPT). Specifically, the graph structure encoder is adopted for structural feature encoding. Then, a structure-guided fusion module with two simple yet effective strategies, i.e., weighted summation and alignment constraint, is designed to inject the structural information into both the textual and visual features. To the best of our knowledge, SGMPT is the first MPT for multi-modal KGR, which mines structural information underlying MKGs. Extensive experiments on FB15k-237-IMG and WN18-IMG, demonstrate that our SGMPT outperforms existing state-of-the-art models, and proves the effectiveness of the designed strategies.

Main paper: 
 introduction: 
 Multi-modal knowledge graphs (MKGs), which intuitively organize information in various modalities, can benefit many practical downstream tasks, such as recommendation systems [20, 52] , information retrieval [13, 49] , and visual question answering [21] . Specifically, compared to traditional KGs, extra multi-modal data are linked with entities in MKGs to provide more meaningful information, such as visual and textual attributes, which makes them closer to the real world. However, existing multi-modal KGs may suffer from even more severe incompleteness issues due to the insufficient accumulation of multi-modal corpus. It compromises their usability and impairs their effectiveness. To address the limitations, more and more recent attempts [7, 44, 45, 47, 54, 55] for knowledge graph reasoning (KGR) are taken in multi-modal scenarios, and our work also lays in this scope.The incompleteness issues, as a common nature among different types of KGs, have been widely studied these years [28] . According to previous attempts for KGR, we can easily draw a conclusion that the performance of the KGR model highly depends on whether the information in KGs is fully leveraged or not. For example, the conventional KGR models [9, 38, 42, 51] focus on mining the structural information in static KGs. However, an additional effective mechanism for temporal information is important for these models to achieve better reasoning in temporal KGs. It is similar to multimodal knowledge graph reasoning (MKGR), which requires specific mechanisms to mine the multi-modal information since previous unimodal KGR models can also not scale well to multi-modal scenarios. Following this trace, different mechanisms are designed based on the unimodal KGR models to fuse the heterogeneous multi-modal features of each entity into a unified representation, which builds up various multi-modal KGR models. For example, IKRL [46] and MTRL [31] are both developed based on TransE [4] but with different fusion strategies. The former model adopts the attention mechanism to integrate the visual information, while the fusion module in the latter one offers three different fusion options. However, with the development of general artificial intelligence (AGI), the pre-trained transformer architecture has drawn increasing attention as a more general and powerful paradigm, especially for multi-modal scenarios. Inspired by their success in other fields, lots of pre-trained transformer models for KGR have come out these years, such as KG-BERT [50] , KGformer [29] , and etc. However, the research on developing an effective multi-modal pre-trained transformer (MPT) for KGR is still at an early stage, which leaves us a huge gap to explore.Among the MPT models for KGR, MKGformer [7] is the most representative model with the best reasoning capacity. Although achieving promising performance for MKGR, it still fails to fully leverage the structural information underlying the knowledge graph, which hinders their reasoning capacity. Unlike other multi-modal data, MKG usually contains three types of information, i.e., textual information M 𝑡 (e.g., text description), visual information M 𝑣 (e.g., images), and graph structural information G as shown in Fig. 1 (a) . MPT models for other multi-modal scenarios only take consideration of the first two modalities, which is also how MKGformer does for MKGR task as shown in Fig. 1 (b) . More specifically, the graph structure of MKG is only utilized as a retrieval map for matching images and text descriptions corresponding to the same entity. In this manner, the rich structural information underlying the graph structure G is ignored, such as the relational information between different entities and the topological information within the graph structure. This structural information will definitely benefit the expressive ability of the models, which has been proven in those multi-modal non-transformer KGR models [44, 54] . As it is currently ignored by MPT models for MKGR, all we need is to design an effective fusion mechanism to mine such structural information, which will endow the existing MPT models with greater capacity for better reasoning performance.Following this idea, we propose a novel graph Structure Guided Multi-modal Pre-trained Transformer model for knowledge graph reasoning, termed SGMPT, by designing a plug-and-play mechanism to leverage the structural information omitted by previous MPT models as shown in Fig. 2 . More specifically, the graph structure encoder is adopted for structure feature encoding. Then, a structure-guided fusion module with two different strategies, i.e., weighted summation and alignment constraint, is first designed to fuse the structure information in both the textual and visual features. Concretely, (1) weighted summation directly adds the generated structural embedding with the textual and visual embeddings in the segment for the entity. Besides, (2) the alignment constraint adopts the alignment loss, i.e., MSE loss [15, 39] , to guide the learning procedure by refining original textual and visual embeddings according to structural information. Moreover, the above strategies can be adopted both individually and composedly. To the best of our knowledge, SGMPT is the first multi-modal pre-trained transformer for KGR, which tries to mine the structural information underlying the knowledge graph. In addition, extensive experiments are carried out on two typical benchmark datasets to demonstrate the promising capacity of SGMPT from four aspects, i.e., superiority, effectiveness, efficiency, and sensitivity. The main contributions are summarized below:• We propose a novel and simple graph structure guided multimodal pre-trained transformer model for knowledge graph reasoning, termed SGMPT, by effectively making use of the knowledge graph structural information. • We adopt the structural encoder and design a plug-and-play mechanism, i.e., structure-guided fusion module, is proposed to complement the omitted graph structural information for MPT models for KGR. Specifically, the structure-guided fusion module contains two different strategies, i.e., weighted summation and alignment constraint, which can be adopted both individually and composedly.• Extensive experiments on FB15k-237-IMG and WN18-IMG datasets demonstrate the capacity of SGMPT from four aspects, i.e., superiority, effectiveness, efficiency, and sensitivity, and also proves the effectiveness of the structural information. The codes will be open-sourced after the review procedure. 
related work 2.1 multi-modal knowledge graph reasoning: 
 Multi-modal knowledge graph reasoning (MKGR) aims to infer the potential missing facts in multi-modal knowledge graphs, which can be roughly divided into two types, i.e., non-transformer models and transformer models, according to the model architectures.2.1.1 Non-Transformer Multi-modal KGR Models. Most of the MKGR models are developed based on non-transformer architectures. Different mechanisms are designed to encode the extra modal information by extending the original unimodal KGR models, such as TransE [4] . For example, IKRL [46] first adopts the attention-based mechanism to integrate the visual information and the original structural information generated by the translation-based KGR models. MTRL [31] offers three different strategies, i.e., simple summation, DeViSE [16] , and Imagined [10] to integrate multi-modal information. In addition, TransAE [44] utilized an auto-encoder to use them. Moreover, MoSE [54] exploits three ensemble inference techniques to combine the modality-split predictions by assessing modality importance. Recently, RSME [43] designed a forget gate with an MRP metric to select valuable images for multi-modal KGR, which tries to avoid the influence caused by the noise from irrelevant images corresponding to entities. However, our model does not belong to this type.Multi-modal KGR Models. The transformer models originated form natural language processing [5, 12, 33] , and quickly shifted the paradigm of image processing [3, 14] from fully supervised learning to pre-training and fine-tuning. Due to their promising capabilities in multi-modal scenarios, various general multi-modal pre-trained transformer (MPT) models [8, 18, 24, 26, 30, 35, 37] have been proposed these years. However, the target optimization objects of the above multi-modal pre-trained models are less relevant to knowledge graph reasoning (KGR) tasks. Due to the variance between the multi-modal knowledge graph (MKG) and other multi-modal data, directly applying the above general MPT models to multi-modal knowledge graph reasoning (MKGR) may not lead to good reasoning [7] . Meanwhile, pre-trained transformer models [6, 7, 19, 23, 25, 29, 36] for KGR are also springing up, such as KG-BERT [50] , which is the first pre-trained contextual language model for the KGR task, etc. However, the research on developing an effective multi-modal pre-trained transformer (MPT) for KGR is still at an early stage. Among them, MKGformer [7] is the most representative attempt with promising reasoning capacity, which leaves us huge space to explore better MPT models for KGR. But MKGformer [7] also ignores the key difference of the characteristic between MKG and other multi-modal data. Specifically, unlike other multi-modal data, MKG usually contains three types of information, i.e., textual information M 𝑡 (e.g., text description), visual information M 𝑣 (e.g., images), and graph structural information G as shown in Fig. 1 (a) . MPT models for other multi-modal scenarios only take consideration of the first two modalities, which is also how MKGformer does for MKGR task as shown in Fig. 1 (b) . In other words, the graph structure of MKG is only utilized as a retrieval map for matching images and text descriptions corresponding to the same entity. In this manner, the rich structural information underlying the knowledge graph G is ignored, such as the relational information between different entities and the topological information within the graph structure. This structural information will definitely benefit the expressive ability of the models, which has been proven in those multi-modal non-transformer KGR models [44, 54] . As it is currently ignored by MPT models for MKGR, all we need is to design an effective fusion mechanism to mine such structural information for multi-modal transformer KGR models. To this end, our work endows the existing MPT models with greater capacity for those omitted structural information to achieve better reasoning.Information fusion aims to integrate information from different modalities to contribute to the specific downstream tasks [2] , which is usually treated as one of the important step for multi-modal and multisource tasks. There are generally two ways to fuse the features, i.e., (1) combining every single modal representation in its own feature space, such as summation, average pooling [1, 40, 56] , and (2) learning the unified representations by projecting different modal representations into the same latent space based on a well-designed objective function [31, 32, 46] . Inspired by them, our SGMPT is the first MPT model to fuse the structural information with the original textual and visual information by introducing a plug-and-play structure-guided fusion module. Two strategies are designed in the structure-guided fusion module, including weighted summation and alignment constraint. Concretely, weighted summation belongs to the first type, while the alignment constraint belongs to the second type. Both the above two types of strategies can be adopted individually and composedly. More details about the fusion strategies are illustrated in Section III.C.(2). 
method: 
 In this section, we will introduce the details of the proposed graph structure guided multi-modal pre-trained transformer model, termed SGMPT, from three aspects, i.e., preliminary, multi-modal pretrained transformer backbone, and guidance of structure information. The overall framework of our SGMPT is shown in Fig. 2 , and the pseudo-code of our SGMPT is shown in Algorithm 1.In this section, we introduce prior knowledge for the multi-modal knowledge graphs and the reasoning task over them.The multi-modal knowledge graph is defined as a directed graph MKG = (E, R, G, A M ), where E and R represent the entity set and the relation set respectively, andG = {(𝑒 ℎ , 𝑟 ℎ,𝑡 , 𝑒 𝑡 ) | 𝑒 ℎ , 𝑒 𝑡 ∈ E, 𝑟 ℎ,𝑡 ∈ R}is the set of fact triplets. A M represents the set of multi-modal attributes corresponding to each entity, which contains two types of modalities, i.e., text descriptions (M 𝑡 ) and image descriptions (M 𝑣 ).As for the reasoning task over multi-modal KGs, it is defined in this paper as follows. Given the missing facts (𝑒 ℎ , 𝑟 ℎ,𝑡 , ?), the main goal of knowledge graph reasoning (MKGR) is to infer the entity 𝑒 𝑡 based on the MKG. Similar to previous multi-modal pretrained transformer (MPT) KGR models [7] , the MKGR tasks in this paper can be divided into two sub-tasks, including: (1) Pretraining: image-text incorporated entity representation learning, and (2) Fine-tuning: relation reasoning over multi-modal entity representations. Both of the sub-tasks are trained on multi-modal knowledge graph datasets. As shown in Eq. ( 1) and Eq. ( 2), the [CLS] and [SEP] are two separation symbols, and [MASK] is the prediction symbol. Notably, the multi-modal attributes, i.e., visual and structural features, are encoded and integrated into the textual feature, and both pre-training and fine-tuning tasks are still reformulated as masked language modeling (MLM) tasks based on the textual features.Pre-training. The pre-training procedure aims to match the multimodal attributes with the corresponding masked entity 𝑒 𝑖 .EQUATIONFine-tuning. For reasoning task 𝑇 (𝑒 ℎ , 𝑟 ℎ,𝑡 , ?), the main goal of the fine-tuning procedure is to predict the masked target entity 𝑒 𝑡 .𝑇 (𝑒 ℎ , 𝑟 ℎ,𝑡 , ?) = [CLS] 𝑒 ℎ 𝐴 𝑡 ℎ [SEP] 𝑟 ℎ,𝑡 [SEP] [MASK] [SEP] (2)The MKGformer [7] , which is the most representative MPT model for KGR, is selected as our MPT backbone. and outputs the textual feature H 𝑡 .EQUATIONVision Encoder. Vision encoder 𝑓 𝑣 (•) is composed of the first 𝐿 𝑣 layers of ViT [14] , which aims to capture basic visual features from the patched images. It takes images M 𝑣 as input, and outputs the visual feature H 𝑣 .EQUATIONMulti-modal Information Encoder. Following [7] , multi-modal information encoder 𝑓 𝑚 (•) aims to model multi-modal features of the entity across the last 𝐿 𝑚 layers of ViT and BERT with multi-level fusion. It takes learned representations from previous encoders as input, and outputs the multi-modal representations for inference.To guide the learning procedure for the multi-modal pre-trained transformer (MPT) model for KGR with the structural information, two novel modules are proposed, i.e., structure encoder and structure-guided fusion module. The structure encoder aims to encode the structural information into the feature vector, and the structure-guided fusion module tends to fuse the omitted structural information into the existing MPT models for KGR. [53] , ComPGCN [42] , Nodepiece [17] , etc. This paper selects one of the state-of-the-art models, HAKE [53] , as the structure encoder to generate the structural features for most of the experiments. In addition, we also present the influence of different structure encoders in Sec. 4.3.2.EQUATIONThe structure-guided fusion module is designed to fuse the structural information. Two simple yet effective strategies are designed in the structure guide, i.e., weighted summation, and alignment constraint, which can be adopted both individually and composedly (See Fig. 2 ). The details of each strategy will be illustrated as follows.Weighted Summation. Weighted summation combines the generated structural embedding with the segment in textual and visual embeddings corresponding to the entity in its own feature space, as shown in Fig. 3 (a). The summation procedure can be divided into two parts, i.e., (1) text-structure feature generation and (2) vision-structure feature generation. More specifically, for the first part, we first extract the specific token representation vector h s e h ∈ R 1×D from the structural feature matrix H s ∈ R N×D for each entity. Then, the fused feature vector is generated according to Eq. ( 4).EQUATIONwhere 𝜆 𝑡𝑠 𝑠 is the weighted hyper-parameter for text-structure feature generation. After that, we replace the feature vector h ts e h for h t e h in the original textual feature matrix and generate the text-structure feature matrix H ts . As for the second part, we first expand structural feature vector h s e h ∈ R 1×D to the structural feature matrix H s e h ∈ R I×D based on the Eq. ( 8) below.EQUATIONwhere H s e h has the same dimension as the visual feature matrix H v e hfor each entity and 1 I is the row vector with all elements set to 1. This operation aims to expand the vector h ∈ R 1×D to matrix H ∈ R I×D . For example, given h as [1, 2, 3], we can generate the feature matrix H as [1, 1, 1; 2, 2, 2; 3, 3, 3]. Extract the structural vector h s e for corresponding entity 𝑒 from 𝐻 𝑠 , and expand it to the matrix H s e by Eq. ( 7); 5:Generate the textual representation H t e for target entity by Eq. ( 3); 6:Extract the token vector h t e from H t ; 7:Generate the visual representation H v e for target entity by Eq. ( 4);8:Generate the text-structure vector h ts e h by Eq. ( 6).9:Obtain the text-structure feature H ts by reassigning h ts e h to the corresponding position in H t . 10:Generate the vision-structure feature H vs e h (i.e., H vs ) by Eq. ( 8). Calculate alignment losses L 𝑡𝑠 and L 𝑣𝑠 by Eq. ( 9) and Eq. ( 10);13:Calculate cross-entropy loss L 𝑐𝑒 referred to [7] ;14:Optimize the whole network 𝑊 by minimizing L in Eq. ( 12); 15:end for 16: end for Then, the fused vision-structure feature matrix H vs is generated according to Eq. ( 8).EQUATIONwhere 𝜆 𝑣𝑠 𝑠 is another weighted hyper-parameter for vision-structure feature matrix generation.Alignment Constraint. Alignment constraint assists in learning unified representations by aligning the textual and visual feature representations to the structural feature representation using a selfsupervised alignment loss, i.e., MSE loss [15, 39] . The loss helps pull together different embeddings from different modalities. Similar to the weighted summation, the alignment constraint can also be divided into two parts, i.e., (1) text-structure feature alignment and (2) vision-structure feature alignment.Concretely, two types of loss functions are adopted, including text-structure alignment loss L 𝑡𝑠 and vision-structure alignment loss L 𝑣𝑠 .L 𝑡𝑠 = MSE(h t e h , h s e h ) = h t e h ∥h t e h ∥ 2 - h swhere ∥ • ∥ denotes the 2-norm. Our network is optimized by minimizing the contrastive loss and H s e h is generated by Eq. ( 8). Moreover, the total loss L 𝑎 for alignment is shown below:EQUATIONwhere 𝜆 𝑡𝑠 𝑎 and 𝜆 𝑣𝑠 𝑎 are two trade-off hyper-parameters. With the additional alignment losses, the final objective function equals the sum of the cross-entropy loss L 𝑐𝑒 with the alignment losses L 𝑎 :EQUATION)where the cross-entropy loss is used as the original task loss for link prediction.In this section, we discuss the computational complexity of our method. According to Algorithm 1, the training process can be divided into several parts: (1) Initialization, ( 2) Training, and (3) Optimization. We first discuss the complexities of these parts, respectively. We denote the epoch number as 𝑡, the dimensional size of representations as 𝐷, the number of entities as E, the number of images for each entity (visual attribute) as 𝐼 , the text tokens number for each entity (textual attribute) as 𝑇 . The total complexity of our model can be formulated as𝑂 (𝑇 𝐷 + 𝐼𝐷 + 𝑡 E ((𝐼 2 + 𝑇 2 + 𝐼𝑇 )𝐷 2 + 𝐼𝑇 𝐷 2 + 𝐼 2 𝐷 2 + 𝑇 2 𝐷 2 + 𝐼 2 𝑇 2 )) = 𝑂 (𝑡 E ((𝐼 2 + 𝑇 2 + 𝐼𝑇 )𝐷 2 + 𝐼 2 𝑇 2 )).Note that the hyper-parameter 𝐼 ≤ 10 is a small constant and can be neglected, and in the actual training, 𝑡 ≤ 30,𝑇 ≤ 64, 𝐷 < 1000. Thus, the main complexity can be denoted as 𝑂 (𝑡 E𝑇 2 𝐷 2 ), and this complexity depends mainly on 𝑂 (E), i.e., the scale of the dataset.We emphasize the attributes of SGMPT to highlight our contributions and novelties. As an attempt at pre-trained transformer models for multi-modal knowledge graph reasoning, we are the first to work to integrate the structural information underlying the MKGs.Inspired by multi-modal information fusion strategies for other tasks, we design two fusion strategies, i.e., weighted summation and alignment constraint, to fuse the extra structural information into the original pre-trained transformer models for MKGR. These two strategies can be used either independently or composed.Besides, four trade-off hyper-parameters are introduced to adjust the weights of structural information. Meanwhile, a sensitive analysis is carried out on them. Instead of an independent MKGR model, our SGMPT is more like an auxiliary mechanism to assist existing multi-modal pre-trained transformer models for knowledge graph reasoning in achieving better reasoning performance. 
experiment: 
 In this section, we first introduce the experimental settings from four aspects, including datasets, evaluation metrics, implementation, and compared baselines. Then, we comprehensively analyze the proposed SGMPT by answering the following questions.• Q1: Superiority. Does SGMPT outperform the existing stateof-the-art existing multi-modal knowledge graph reasoning models, especially for the transformer models? • Q2: Effectiveness. Are the adopted structure encoder and structure-guided fusion modules effective in fusing structure We conduct experiments to answer the above questions. Specifically, answers of Q1 to Q4 are offered in Sec. 4.2 to 4.5.4.1.1 Datasets. Two commonly used available datasets, i.e., FB15K-237-IMG and WN18-IMG, for multi-modal knowledge graph relation reasoning are used in this paper. These datasets include three modalities, including text descriptions, corresponding images, and graph structures. Specifically, both FB15k-237-IMG [31] and WN18 [31] datasets are constructed by extending ten images for each entity based on FB15k-237 and WN18. The detailed statistics of these two datasets are shown in Tab. 2.The experiments are implemented on the computer with an Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz, 64GB RAM, and one GeForce RTX 3090 Ti GPU using PyTorch 1.10.0 in CUDA 11.1. The model parameters are optimized using Adam [22] optimizer, and we conduct a grid search to find suitable hyperparameters. We select MKGformer as our backbone. Following it, we adopt the BERT base [12] and ViT-B/32 [14] as the text encoder and vision encoder in SGMPT. Besides, a multi-modal information encoder is chosen as the M-Encoder in [7] . As for the structure encoder, most of our experiments are carried out based on the HAKE [53] , but we also evaluate the performance of our model with other typical structure encoders, including HousE [27] , and COMPGCN [42] . Besides, hyper-parameter 𝜆 𝑡𝑠 𝑠 , 𝜆 𝑣𝑠 𝑠 , 𝜆 𝑡𝑠 𝑎 and 𝜆 𝑣𝑠 𝑎 are set as 0.01, 0.01, 0.001, and 0.001. Following previous works [7, 43] , we use four metrics to evaluate the performance of our model, i.e., (1) Hits@k, where k ∈ {1, 3, 10}, and (2) Mean Rank (MR). Besides, the mean results of three runs of each experiment are reported.The compared models include two types, i.e., non-transformer KGR models and transformer KGR models. Among the models in the first type, there are five unimodal KGR state-of-the-art models, including TransE [4] , DisMult [48] , Com-plEX [41] , ConvE [11] , and RGCN [34] , and three multi-modal state-of-the-art KGR models, including IKRL [46] , TransAE [44] and RSME [43] . As for the transformer models, there are three multimodal models except the KG-BERT [50] , which is the first unimodal KGR model developed based on transformer architecture. Visual-BERT [26] and ViLBERT [30] are the general multi-modal models that can also be applied to multi-modal KGR. Besides, MKGformer is the representative multi-modal KGR model with transformer architecture. However, none of the transformer KGR models makes use of the structure information except for our SGMPT.The overall performance comparison is carried out to answer Q1. We compare our SGMPT with thirteen other state-of-the-art models on two benchmark datasets. The results in Tab.3 show that our SGMPT outperforms all of the evaluation metrics compared to nontransformer KGR models. Besides, compared to the transformer KGR models, our SGMPT outperforms most of the evaluation metrics compared to transformer KGR models. In particular, it is more apparent that our model boosts the performance on FB15k-237-IMG. Concretely, our model makes 4.1%, 7.6%, and 2.6% improvements on Hits@1, Hits@3, and Hits@10 compared to non-transformer KGR models, and 3.7%, 2.8%, and 2.2% improvements on Hit@1, Hits@3, and Hits@10 compared to transformer KGR models, respectively. It shows the superiority of our model SGMPT. Besides, it also indicates the advances of the transformer paradigm for the MKGR task, since the average performances of transformer KGR models are better than non-transformer KGR models in the multimodal scenario. Although the MR values of our model are not the best, they still occupy the second-best positions. Moreover, our MR performances are actually better and comparable compared to the backbone MPT model, i.e., MKGformer. The above results show that our SGMPT can achieve better reasoning performance in the multi-modal scenario, which further indicates the effectiveness of involving the structural information in MPT models.The ablation studies are conducted on FB15k-237-IMG to answer the Q2. Specifically, two subquestions need to be answered in the following subsections, i.e., (1) "Can the structure-guided fusion modules make differences?" and (2) "Will different structure encoders benefit the structure information fusion procedure?". 5 shows that performance boosts are made by the strategies used in the structure-guided fusion module, i.e.,, 5.9%, 3.7%, 2.8%, and 2.2% improvements for MR, Hits@1, Hits@3, and Hits@10. More specifically, the alignment constraint and weighted summation are relatively equivalently effective to MKGR. Besides, vision-structure fusion is less than text-structure fusion, due to the information loss caused by the vector expanding operations. In all, the promising results demonstrate the effectiveness of the module and also prove that strategies can be adopted either individually or composedly.Encoder. We also replace the structure encoder 𝑔(•), i.e., HAKE [53] , as three other typical structure encoders, including CompGCN [42] , and Nodepiece [17] .Experiments are further conducted based on FB15k-237-IMG for Hit@1 and Hit@10 (See Fig. 4 ). It indicates that various structure encoders can all benefit the structure information fusion procedure, i.e., on average 2.5% and 2.0% improvements on FB15k-237-IMG and WN18-IMG separately. Besides, the HAKE is the most effective choice among these three structure encoders.Based on promising results and the above analyses in Sec 4.3.1 and Sec. 4.3.2, we can assert that both the adopted structure encoder and designed structure-guided fusion modules are effective in fusing the omitted structure information in KG into the MPT model for better MKGR performance.We analyze the complexity of our SGMPT from two aspects, i.e., parameter number and the running time of the models. Concretely, the comparison is carried out between our SGMPT and the backbone MKGformer on both FB15k-237-IMG and WN18-IMG. According to Tab. 4, it is observed that the efficiency of our model is a little bit worse than MKGformer, i.e., average 0.45% and 3.3% increasing on the number of parameters and running time, respectively. Since the structure fusion mechanism is designed to complement the omitted structure information, our SGMPT will definitely be more complex and time-consuming. However, considering the performance improvements for MKGR, it is acceptable with the comparable complexity, which indicates that our SGMPT does not raise unnecessary parameters and running time.We investigate the influence of the hyper-parameter 𝜆 𝑡𝑠 𝑠 , 𝜆 𝑣𝑠 𝑠 , 𝜆 𝑡𝑠 𝑎 , 𝜆 𝑣𝑠 𝑎 on FB15k-237-IMG for Hit@1 and Hit@10. Besides, the scope of these four hyper-parameters is selected in {0.001, 0.01, 0.1, 1}. We observe that the MKGR performance will not fluctuate greatly when the parameter is varying in Fig. 5 . It demonstrates that the performance of SGMPT is insensitive to these hyper-parameters. We can further find out that best performances are reached with the combination of 𝜆 𝑡𝑠 𝑠 , 𝜆 𝑣𝑠 𝑠 , 𝜆 𝑡𝑠 𝑎 and 𝜆 𝑣𝑠 𝑎 set to (0.01, 0.01, 0.001, 0.001). 
conclusion: 
 In this paper, we propose a novel and simple graph Structure Guided Multi-modal Pre-trained Transformer model for knowledge graph reasoning, termed SGMPT. As the first multi-modal pre-trained transformer model to leverage the structure information for multimodal knowledge graph reasoning, SGMPT is the first work to add a specific structure information fusion procedure based on the MPT backbone. The structure encoder and structure-guided fusion module are required to complete the procedure. More concretely, various KGE models can be selected as the structure encoder, and two different strategies, i.e., weighted summation and alignment constraint, can be adopted both individually and composedly. Extensive experiments on two benchmark datasets, i.e., FB15k-IMG and WN18-IMG, demonstrate the promising capacity of our SGMPT from four aspects, including superiority, effectiveness, complexity, and sensitivity.In the future, we plan to continue to improve the capacity of our model toward its limitations. For example, four hyperparameters are enrolled to complement the omitted structural information. Moreover, the weighted summation strategy can be more fine-grained, though the current simple addition can also achieve promising improvements.  
: 
 Figure 1 : Illustration of multi-modal knowledge graph and existing pre-trained transformer KGR models. M 𝑡 , M 𝑣 , and G represent the textual information, visual information, and graph structural information, respectively. Besides, BERT [12] and ViT [14] are two commonly used transformers for encoding M 𝑡 and M 𝑣 , and structure-guided fusion module (S-Guider) is a novel module designed in our SGMPT to leverage G. Note that 𝑒 and 𝑟 denote entity and relation separately. 
 
 Appendix: Submission ID: 2615. 2024-04-10 06:13. Page 3 of 1-9.Submission ID: 2615. 2024-04-10 06:13. Page 4 of 1-9.Submission ID: 2615. 2024-04-10 06:13. Page 6 of 1-9.Submission ID: 2615. 2024-04-10 06:13. Page 7 of 1-9.Submission ID: 2615. 2024-04-10 06:13. Page 8 of 1-9.Submission ID: 2615. 2024-04-10 06:13. Page 9 of 1-9. 


Decision: In this paper, the authors have proposed a novel and simple graph Structure Guided Multi-modal Pre-trained Transformer model for knowledge graph reasoning, termed SGMPT. All the reviewers recognize the contributions/novelties of this paper. However, they have pointed out some limitations. The authors are expected to address these limitations in the camera ready version.

Reviews: review: paper_summary: paper_summary: This paper introduces a structure guided multimodal pre-trained Transformer model called SGMPT for knowledge graph inference tasks. Unlike previous knowledge graph inference models, this model can utilize the structural information in the knowledge graph and inject structural information into text and visual features by introducing structural encoding and structural guidance fusion modules. The experimental results show that the model performs better than the existing best models on the FB15k-237-IMG and WN18-IMG datasets, and demonstrate the effectiveness of the design strategy.
strength_weakness: strength_weakness: Strengths:
1. Traditional multi-modal pre-training models only consider the relationship between textual and visual information, overlooking the significance of structural information. The SGPMT model, however, introduces structural information and employs a structure-guided fusion module, enabling the model to better leverage multimodal information and enhance its performance. SGMPT stands as the first multi-modal pre-training Transformer designed for Knowledge Graph Reasoning (KGR).

2. SGMPT designs two effective structure-guided fusion strategies: weighted summation and alignment constraint, which can be adjusted according to the demands of different tasks, thereby further enhancing the model's adaptability and generalization capability.

3. Acting as a plug-and-play fusion mechanism, SGMPT allows for the flexible selection of various Knowledge Graph Embedding (KGE) models as structural encoders.

4. Experimental results demonstrate that SGMPT achieves superior performance across two public datasets, affirming the efficacy and superiority of the SGMPT model.
limitations: limitations: 1.The representation in Figure 5 is not very clear and could be replaced with a more effective visualization method.

2.Although a simple additive approach currently yields noticeable improvements, the weighted summation strategy appears rather rudimentary. 

3.Ideally, illustrating the attention weights assigned to the text-structure features and vision-structure features after their fusion with structure information would provide a clearer understanding of precisely how the incorporated structural information strengthens the connections between entities.
paper_summary: paper_summary: This paper introduces a new model called Structure Guided Multi-modal Pre-trained Transformer (SGMPT) for knowledge graph reasoning. It addresses the limitations of previous multi-modal pre-trained transformers (MKGformer) by fully utilizing the rich structural information present in multi-modal knowledge graphs .
SGMPT incorporates a graph structure encoder to encode structural features. It also includes a structure-guided fusion module that uses weighted summation and alignment constraint strategies to inject the structural information into both textual and visual features.
strength_weakness: strength_weakness: SGMPT addresses the limitation of previous multi-modal pre-trained transformers by fully utilizing the rich structural information present in multi-modal knowledge graphs. 
It incorporates a graph structure encoder and a structure-guided fusion module to inject the structural information into both textual and visual features.
limitations: limitations: 1. The contribution of this paper is limited, as it is an incremental work based on MKGformer. The basic framework of this paper follows MKGformer completely, with the addition of a graph-guide branch.

2. One curious point is that while MKGformer performs multiple tasks (multimodal link prediction, multimodal RE, and multimodal NER), why this paper only focuses on link prediction?

3. Where does the viewpoint come from that "Among the MPT models for KGR, MKGformer is the most representative model with the best reasoning capacity"? Its performance is not better than some of the latest methods, so how is the "best reasoning capacity" evaluated?

4. Why is there such a significant difference between the reported experimental results of MKGformer in the FB15k-237-IMG dataset (Hits@10 = 0.573) and the results in this paper (Hits@10 = 0.499)?

5. The paper lacks some necessary citations and comparison experiments. It fails to compare the proposed method with the latest state-of-the-art methods. It would have been beneficial to include comparisons with recent approaches such as QuatE [1] for uni-modal KGC, CSProm-KG [2] and SimKGC [3] for description-based KGC, and LAFA [4], MMNER [5], AdaMF-MAT [6] for multimodal KGC.

6. The three-dimensional histogram in Figure 5 makes it difficult to extract information and judge the effectiveness of different parameters. It is recommended to replace it with a heatmap.




[1] Quaternion Knowledge Graph Embeddings 

[2] Dipping PLMs Sauce: Bridging Structure and Text for Effective Knowledge Graph Completion via Conditional Soft Prompting

 [3] Simkgc: Simple contrastive knowledge graph completion with pre-trained language models

 [4] LAFA: Multimodal Knowledge Graph Completion with Link Aware Fusion and Aggregation 

[5] Relation-enhanced negative sampling for multimodal knowledge graph completion

 [6] Unleashing the Power of Imbalanced Modality Information for Multi-modal Knowledge Graph Completion
paper_summary: paper_summary: This paper introduces a novel method named SGMPT (graph Structure Guided Multi-modal Pre-trained Transformer) with a graph structure encoder and structure-guided fusion module. The method is simple yet effective, which outperforms existing state-of-the-art methods on two multimodal knowledge graph reasoning datasets FB15k-237-IMG and WN18-IMG.
strength_weakness: strength_weakness: 1. The idea that mines structure information for multimodal knowledge graph reasoning is reasonable and effective intuitively.
2. The proposed structure-guided fusion module fuses structural information with modal information in a very simple and intuitive way.
3. The results on two multimodal knowledge graph reasoning datasets FB15k-237-IMG and WN18-IMG demonstrate the effectiveness of the method.
4. The ablation study and analysis are comprehensive and specific, and the experimental organization is reasonable.
limitations: limitations: 1. The text encoder and the visual encoder are relatively simple. It would be interesting to experiment with larger models, such as LLaVA.
2. The initialization of the multi-modal information encoder is important. What is used to initialize it?
paper_summary: paper_summary: A graph structure-guided multimodal pre-training framework using the Transformer model for knowledge graph reasoning is proposed to supplement the omitted graph structure information used for knowledge graph reasoning with a structure encoder and a structure-guided fusion module.
strength_weakness: strength_weakness: 1. By introducing a graph structure encoder and a structure-guided fusion module, the structural information in the knowledge graph is utilized for the first time to improve the inference ability of the model.

2. The design is simple but effective, fusing structural information into textual and visual features through two strategies, weighted summation and alignment constraints, to improve the inference performance of the model.
limitations: limitations: 1. The performance improvement of the experimental results is not very high, not very convincing, and even less than 1% on the WN18-IMG dataset.

2. Whether the baseline for comparison of experimental results is available for the last two years of the method.