Multimodal Contextual Interactions of Entities: A Modality Circular Fusion Approach for Link Prediction

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-09 13:07:31

Abstract: Link prediction aims to infer missing valid triplets to complete knowledge graphs, with recent inclusion of multimodal information to enrich entity representations. Existing methods project multimodal information into a unified embedding space or learn modality-specific features separately for later integration. However, performance was limited in such studies due to neglecting the modalities compatibility and conflict semantic carried by entities in valid and invalid triplets. In this paper, we aim at modeling inter-entity modality interactions and thus propose a novel modality circular fusion approach (MoCi), which interweaves multimodal contextual of entities. Firstly, unlike most methods in this task that directly fuse modalities, we design a triplets-prompt modality contrastive pre-training to align modality semantics beforehand. Moreover, we propose a modality circular fusion model using a simple yet efficient multilinear transformation strategy. This allows explicit inter-entity modality interactions, distinguishing it from methods confined to fuse within individual entities. To the best of our knowledge, MoCi presents one of the pioneering frameworks that tailored to grasp inter-entity modality semantics for better link prediction. Extensive experiments on seven datasets demonstrate our model yields SOTA performance,  confirming the efficacy of MoCi in  modeling inter-entity modality interactions. Our code is released at https://github.com/MoCiGitHub/MoCi.

Main paper: 
 related work 2.1 uni-modal link prediction: 
 Uni-modal link prediction utilizes the structural topology of KG and embeddings to predict missing links. Models like RotatE [47] interpret relations through complex space rotations, effectively handling various relational properties, while ANALOGY [11] maintains analogical consistency to boost reasoning. Additional models include TransE [3] , DistMult [4] , ComplEx [31] , GC-OTE [40] , TuckER [14] , ConvE [30] , PairRE [17] , MDCN [44] , WGCN [7] , and JointE [45] , each offering distinct approaches to link prediction. The multimodal information of entities is not taken into account in these methods.Entity-level modality aggregation models improve link prediction by incorporating multimodal information such as visual images and text description to enrich entity representations. For example, HRGAT [27] uses a hierarchical graph attention network to integrate modalities effectively, while IKRL [25] combines images with structured data to improve visual entity recognition. Additional models like MKGformer [32] , OTKGE [49] , MKBE [22] , QEB [33] , CMGNN [23] , TBKGC [13] , RSME [20] , TransAE [48] , VBKGC [42] , MULTIFORM [36] and AdaMF-MAT [43] also utilize diverse modalities but often struggle to preserve the unique properties of each modality when merging them into a single embedding space.Modality split and ensemble models focus on learning modality specific representations and integrating predictions from different modalities in knowledge graphs. MoSE [39] pioneered this framework by introducing an ensemble method that dynamically assesses and integrates the importance of different modalities during inference. Building on this foundation, the IMF [34] model enhances the approach by using a bilinear pooling module. Nonetheless, such models usually restrict the interactions to within individual entity.The performance was limited in such studies due to neglecting the modalities compatibility and conflict semantics carried by entities in valid and invalid triplets. The introduction of MoCi aims to address this deficiency. It presents the first framework tailored to capture inter-entity modality semantics for significantly improving link prediction.  
definitions: 
 A knowledge graph is formalized as 𝐺 = {𝐸, 𝑅,𝑇 }, where 𝐸, 𝑅 denotes entity set and relation set, and 𝑇 = {(ℎ, 𝑟, 𝑡)|ℎ, 𝑡 ∈ 𝐸, 𝑟 ∈ 𝑅} refers to triplets that describe relations between entities. In multimodal KG, each entity is associated with multi-modal context, including textual, visual, and structural information. We define the set of modalities as 𝑀 = {𝑠, 𝑣, 𝑡 }, where 𝑠,𝑣,𝑡 denote structural, visual, and textual, respectively. The objective of link prediction is to infer missing triplets according to the existing factual triplets 𝑇 and multi-modal contexts, which could be generally represented as (?, 𝑟, 𝑡) or (ℎ, 𝑟, ?), namely predicting the possible subject or object. Notations and their definitions are given in Table 1 . 
methodology: 
 In this section, we offer a comprehensive introduction to our proposed models, which is designed to capture inter-entity modality semantics to enhance link prediction. This includes modality-specific embeddings initialization, triplets-prompt modality contrastive pretrain, modality circular fusion, and relational context-aware prediction. The overall framework of the proposed model is illustrated in Figure 3 .Distinguishing from traditional neural networks that directly take raw data features as input and process them through large-scale nonlinear networks, we introduce a novel strategy here: the features of each modality are defined as learnable parameters, and the corresponding parameter embedding is initialized by the entity modality features. In this way, we effectively avoid the model's excessive dependence on deep nonlinear networks while graph is highly sparse. Additionally, this initialization method better retains and utilizes the original feature information of each entity, which encourage the model converge to the appropriate solution faster. In this subsection, we explore the initialization strategies for the textual, and visual, structural modalities as follows:Textual Modality: Entity text descriptions usually contain some valuable textual information, typically including name, type, attribute etc. We utilize the BERT-Base [5] model pre-trained on a large-scale corpus to capture word contextual semantic relations and generate high-quality text feature representations on diverse datasets, such as YAGO15K [29] , VTKG-I [16] and MKG-Y [9] . This approach provides more accurate textual semantic descriptions for subsequent tasks.Visual Modality: For the visual description of each entity, the pretrained VGG16 [28] model is utilized for feature extraction on YAGO15K and DB15K datasets. Considering the diverse complexity of the visual content of different datasets, such as VTKG-I, VTKG-V, and WN18RR++, the Vision Transformer (ViT [2] ) model is adopted to generate more expressive feature representations, which could effectively capture dependencies between different regions of given complex images through their self-attention mechanism.Structural Modality: KG is essentially a topological structure that reflects the connection relation between known entities. Different from the visual and textual attributes of the entity itself, the structural features could further reflect the association characteristics between different entities. Hence, we employ a multi-relational graph convolutional network (MR-GCN [46] ) to learn the graph structure and obtain the initial node embeddings.The embeddings for textual, visual and structural modalities are initialized with features extracted from models ideally suited to their respective characteristics. This detailed strategy is further discussed in Section 5.1, outlines our experimental setup and the specific models employed for link prediction tasks.Pre-trainMost existing methods typically directly fuse different modalities [9, 12, 22, 27, 34, 39] without accounting for potential semantic differences between them. We propose a pre-training strategy before modal feature fusion. The purpose of this strategy is to align modality semantics in advance and interact modality information between entities.Triplets-Prompt Modality Contrastive Learning: In this section, we introduce a novel cross-modal contrastive learning in the context of triplets, which utilizes the semantic compatibility/incompatibility carried by valid/invalid triplets. Concretely, this strategy focuses on triplet-prompt modal contrastive learning, and carefully selects high-confidence positive samples from the existing factual (valid) triplets and negative samples from the invalid context. This strategy also allows for efficient learning of graph structure by leveraging triplets without the need for costly inclusion of all nodes.Formally, the interaction patterns of entities in valid triplets (ℎ, 𝑟, 𝑡) and invalid triplets (ℎ, 𝑟, 𝑡 ′ ), i.e. 𝑠𝑖𝑚 and 𝑑𝑠𝑖𝑚 are modeled as follows:EQUATION)where 𝑝, 𝑞 ∈ 𝑀 are different modalities within the modalities set 𝑀, which includes structural (𝑠), textual (𝑡), and visual (𝑣) modalities. Additionally, a Norm&Activate to align inter-entity modality semantics, refining our model's contrastive learning dynamic tradeoff. Cosine similarity (𝑐𝑜𝑠) is defined as follows:Entity Structure Visual Text Joint 𝐸 " 𝐸 $ 𝐸 # 𝐸 ' 𝑅 " 𝑅 $ 𝑅 # 𝑅 ' Entity Embedding Relation Embedding Relation 𝐸 " 𝐸 $ 𝐸 # 𝐸 $ 𝐸 # 𝐸 " 𝐸 # 𝐸 " 𝐸 $ 𝐸 " 𝐸 $ 𝐸 # 𝐸 $ 𝐸 # 𝐸 " 𝐸 # 𝐸 " 𝐸 $ 𝐸 " 𝐸 $ 𝐸 # 𝐸 $ 𝐸 # 𝐸 " 𝐸 # 𝐸 " 𝐸 $ B a t c hEQUATIONBuilding on the above, we introduce the overall triplet contrastive loss L 𝑐𝐿 as follows:EQUATION) where 𝑂 = {(𝑝, 𝑞) | 𝑝, 𝑞 ∈ 𝑀 } represents all modality combinations, and 𝑁 is the total number of triplets.Pre-training Loss: In L 𝑐𝐿 , we have ensured inter-entity modality interaction. Furthermore, to enhance structural learning and achieve clearer structural embeddings, we have incorporated contrastive and structural losses to define our pre-training loss L 𝑝𝑟𝑒 as follows:L 𝑝𝑟𝑒 = L 𝑐𝐿 + L 𝑠 (5) where L 𝑠 = 𝑓 𝑠 (ℎ, 𝑟, 𝑡) is the structural loss function, with implementation details provided in Section 4.4.The highly sparsity of KG triplets directly affects the design of subsequent network architecture, such as 10𝑒 -6 on YAGO15K and DB15K datasets [29] , etc., which is calculated by dividing the total number of known triplets by that of possible triplets. Hence, how to effectively further avoid deepening the nonlinear inference layers under good parameter initialization conditions, becomes the focus of this section. To this end, we define a set of modality-shared linear transformation parameters, and achieve efficient interaction of multi-modal contexts through a circulant fusion strategy based on Tensor product [41, 46] . This allows explicit inter-entity modality interactions, distinguishing it from methods confined to fuse within individual entities.Specifically, we employ a simple yet effective multilinear transformation strategy inspired by the Tensor product, in which each element denotes a tube rather than a scalar in the conventional matrix. The Tensor product of tensor X ∈ R 𝑁 ×𝐷 𝑀 and corresponding linear transformation parameter tensor W ∈ R 𝐷 ×𝐷 ′ 𝑀 is defined as:EQUATIONwhereZ ∈ R 𝑁 ×𝐷 ′, * indicates the Tensor product operator. The 𝑢𝑛𝑓 𝑜𝑙𝑑 (.) operator flattens tensor X ∈ R 𝑁 ×𝐷 𝑀 to a matrix of size 𝑀𝑁 ×𝐷, and 𝑓 𝑜𝑙𝑑 (•) corresponds to its inverse. 𝑏𝑐𝑖𝑟𝑐 (X) ∈ R 𝑀𝑁 ×𝑀𝐷 is the unfolding block circulant matrix of X. For more detailed explanations about the Tensor product, please refer to Reference [41, 46] . In such a paradigm, we formulate the modality circular fusion as:EQUATIONwhere E ∈ R F (𝑚) = E (𝑚) •W (1) +E (𝑚-1) •W ( |𝑀 | -𝑚+2) +. . .+E (𝑚+1) •W (2) (8)This implies that there are |𝑀 | different combination strategies betweenE (𝑖 ) ∈ R |𝐸 | × |𝐷 | and W ( 𝑗 ) ∈ R |𝐷 | × |𝐷 ′ |, determined by the way the block circulant matrix 𝑏𝑐𝑖𝑟𝑐 (E) generated. Hence, the derived embedding tensor F follows a circulant formulation to fuse features from different modalities roundly, and also ensure semantic consistency encoding among different modalities by parameter sharing. Building on the above, the final joint features J are obtained as follows:EQUATIONThe joint features J then undergo normalization and are processed through activation functions.Scalability: Our model MoCi includes three commonly used modalities of entities, i.e. 𝑀 = {𝑠, 𝑣, 𝑡 }. It is straightforward to generalize the proposed method for fusing more modalities by extending 𝑀. Moreover, despite the parameter W being a tensor, its size is merely number of modalities× feature dimension × feature dimension(transformed), where the number of modalities |𝑀 | = 3 in our work. This ensures efficiency and scalability even including more modalities.Explainability: The proposed circular fusion operation goes beyond merely aggregating matrix multiplication results from corresponding frontal slices in the spatial domain. It embodies the essence in Fourier domain since E * W = 𝐼 𝐹 𝐹𝑇 (𝐹 𝐹𝑇 (E)Δ𝐹 𝐹𝑇 (W)), where Δ indicates slice-by-slice matrix multiplication. This implies the linear fusion across modalities in the Fourier domain, thereby offering a nuanced capture of inter-modality correlations and complementarity. For more detailed explanations about the Tensor product, please refer to Reference [41, 46] .Relations as the most essential description of the association between entities. Therefore, our proposed model further considers relations in the prediction phase. We enrich entity representations through aggregate relation context by extend [21] .Aggregate Relation Context (AggRC): In our model, the contextual relation feature c ∈ R 2|𝑅 | is employed to accurately capture the relational topology of the entity-oriented graph. The corresponding feature is defined as follows: EQUATIONEQUATIONwhere 𝑚 ′ ∈ {𝑠, 𝑣, 𝑡, 𝑗 } corresponds to structural (𝑠), visual (𝑣), textual (𝑡) modalities and joint modalities (𝑗) that obtained from the implement of modality circular fusion. Relation embedding R ∈ R 2|𝑅 | ×𝑑 encompasses both forward and reverse relations. The forward relation can be represented as (ℎ, 𝑟 + , 𝑡), while the reverse relation can be represented as (𝑡, 𝑟 -, ℎ).Training Loss: We extentd the score function of the classical ConvE [30] model by aggregating relation context into entity representations. Given a triplet (ℎ, 𝑟, 𝑡), the prediction score is defined as:EQUATION) where b 𝑚 ′ = 𝐴𝑔𝑔𝑅𝐶 (e 𝑚 ′ ) ∈ R 2𝑑 , represents the entities' representations after being enriched by the aggregate relation context (𝐴𝑔𝑔𝑅𝐶). W 𝑚 ′ ℎ ∈ R 2𝑑 ×𝑑 represents the transformation matrix. The convolutional filter is represented by 𝜔, W 𝑓 the is weight of the fully connected layer. 𝜎 denotes a non-linear activation function such as sigmoid or ReLU, which endows the model with the capacity to discern complex, non-linear relations present within the dataset.To optimize the model, a total loss function is defined to include losses from multiple modalities and the joint modalities (𝑗), as shown below:EQUATIONEach loss term L * corresponds to a specific modality, aiming to accurately grasp inter-entity modality semantics for better link prediction. 𝐷𝑎𝑡𝑎𝑠𝑒𝑡𝑠 𝑇 -𝑃𝑀 𝑉 -𝑃𝑀 𝐸 𝑇 𝐸 𝑉 𝑅 𝑇 𝑅 𝑉 YAGO15K BERT-Base VGG16 ✓ ✓ × × DB15K BERT-Base VGG16 ✓ ✓ × × VTKG-I BERT-Base ViT ✓ ✓ ✓ ✓ VTKG-C BERT-Base ViT ✓ ✓ ✓ ✓ WN18RR++ BERT-Base ViT ✓ ✓ ✓ × MKG-W SBERT BEiT ✓ ✓ × × MKG-Y SBERT BEiT ✓ ✓ × × 
experiments 5.1 datasets: 
 Our model was evaluated on seven diverse multimodal publicly available datasets: YAGO15K, DB15K, VTKG-I, VTKG-C, WN18RR++, MKG-W and MKG-Y. YAGO15K and DB15K datasets are obtained from MMKG [29] ; VTKG-I, VTKG-C and WN18RR++ datasets are obtained from VISTA [16] ; MKG-W and MKG-Y datasets are obtained from MMRNS [9] . Table 2 delivers a detailed statistical overview.And the feature extraction models used for each dataset, along with the availability of textual and visual features within entities and relations, are shown in Table 3 . The acronyms 𝑇 -𝑃𝑀 and 𝑉 -𝑃𝑀 denote the models used for textual and visual feature extraction, respectively. The notations 𝐸 𝑇 , 𝐸 𝑉 , 𝑅 𝑇 , and 𝑅 𝑉 specify the availability of textual, visual features for entities (𝐸) and relations (𝑅).To substantiate the effectiveness of our model, we selected 23 prominent link prediction models proposed in recent years as our baselines. For clarity in comparison, these models are categorized into unimodal knowledge graph embedding models and multimodal knowledge graph embedding models.• Unimodal Models: This category includes models such as TransE [3] , DistMult [4] , RotatE [47] , PairRE [17] , GC-OTE [40] , TuckER [14] , ConvE [30] , ComplEx [31] , and ANAL-OGY [11] , which focus exclusively on the structural learning of given KG. • Multimodal Models: This category includes models such as IKRL [25] , MKBE [22] , TransAE [48] , AdaMF-MAT [43] , MMKRL [35] , KBGAN [6] , MMRNS [10] , MKGC [12] , OTKGE [49] , TBKGC [13] , RSME [20] , MANS [38] , MoSE [39] , and IMF [34] , which focus on integrating multimodal data to enhance KG embeddings. Our experiments were conducted on an NVIDIA RTX A6000 GPU with 48GB of RAM, utilizing the PyTorch [1] deep learning framework for implementation. Throughout the training process, we configured the number of training epochs to 1,000, with a batch size of 256, modality embedding dimensions set at 256, and a learning rate of 0.0005. For baseline methods, we relied on both their originally reported results and our reproduction of those results. This experimental setup ensures a rigorous and fair comparison across all evaluated models.MoCi has a significant performance improvement compared with all baseline models, which fully demonstrates the effectiveness of the proposed methods. As illustrated in Tables 4 and 5 , the MoCi model shows state-of-the-art performance on multiple evaluation metrics. In details, on the YAGO15K dataset, MoCi achieves 11.83% improvement in MRR over the current SoTA approaches, along with improvement of 13.49% in H@1, 11.85% in H@3, and 8.32% in H@10. Furthermore, on the VTKG-I dataset, MoCi achieves 4.73% improvement in MRR compared with the suboptimal approaches, along with enhancements of 4.96% in H@1, 6.87% in H@3, and 4.97% in H@10. Moreover, MoCi also shows significant improvement on the other five datasets.Although unimodal models show more and more complex architectural designs to further enhance the ability to capture the relations between entities in KG, they still face performance limitations due to potential structural biases. Different from these methods, our MoCi effectively alleviates the structural bias of KG and improves the performance link prediction by effectively exploiting the interaction of multimodal complementary information. Moreover, it is worth noting that our model can still achieve a competitive performance level with the unimodal methods as shown in Table 6 , even when only structural information is utilized, which further confirms the effectiveness of our model on structure learning.Multimodal models leverage information from diverse modalities, yet the fusion of modality features is often constrained within individual entities, thereby limiting performance. As opposed to such models, our approach further emphasizes learning inter-entity modality interactions in addition to intra-entity modality interactions. This comprehensive interaction approach enables MoCi to effectively model and utilize the commonalities and complementarities between different modalities, leading to significant improvements in link prediction accuracy.To ascertain the importance of each module in MoCi, we conducted a module ablation study by removing certain components: removing the multimodal circular fusion module as w/o MCF, removing triplets-prompt modality contrastive pre-training module as w/o PCL, and removing the aggregate relation context module is denoted as w/o AggRC. The results shown in Tables 4 and 5 demonstrate a marked decrease in performance with the exclusion of each module-underscoring their joint efficacy. Specifically, the w/o PCL variant validates the necessity of prealigning modality semantics before modality fusion, which provides more harmonious modality features for subsequent interaction and fusion. Moreover, the w/o AggRC variant also illustrates the positive impact of considering relational context on link prediction tasks. It is particularly noteworthy that the removal of the Modality Circulant Fusion (MCF) module, which undertakes the main function of inter-entity modality interactions, has led to a significant decrease in performance. This demonstrates the effectiveness of employing simple yet effective multilinear transformations in modeling inter-entity modality interactions.To validate the impact of modal information on link prediction accuracy enhancement, the modality ablation study is conducted, as shown in Table 6 . This involved assessing the contributions of various combinations of modal embeddings, including structural information (S), visual information (V), and textual information (T), and the results clearly indicate that reliance on a single modality results in the least effective performance, while the incorporation of comprehensive multimodal information significantly improves outcomes. This emphasizes our model's proficiency in adeptly capturing multimodal information to elevate the performance of link prediction tasks.To verify the effectiveness of inter-entity modality interactions for link prediction, we carried out experiments as depicted in Table 7 .The experiments examined whether inter-entity modality interactions during the pre-training and training phases make a difference within MoCi framework.• ×: Indicates scenarios without inter-entity modality interactions, focusing solely on intra-entity modality interactions. • ×(𝐴𝑑𝑑): Signifies the additive fusion of modalities.• ×(𝐵𝑃): Signifies the use of bilinear pooling used in IMF [34] for the fusion of modalities. • (✓): Symbolizes PCL in Pre-Cross, MCF in Cross. The experimental results indicate that, whether during pre-training or the training phase, results with inter-entity modality interactions consistently outperform those without such interactions. This not only validates the efficacy of our model's multimodal fusion approach but also underscores the significance of inter-entity modality interactions in enhancing link prediction accuracy.To highlight the effectiveness of our MoCi model, we employ 𝑡-𝑆𝑁 𝐸 for dimensionality reduction to visually represent the contextual embeddings of football players across different football clubs. Figure 4 shows that representations based solely on unimodal data tend to overlap, reflecting the inherent biases of this approach. In contrast, MoCi leverages circulant interactive multimodal fusion, effectively capturing the inter-entity modality semantics. This method demonstrates MoCi's advanced ability to discern complex inter-entity modality interactions, distinguishing football players' representations by their respective clubs more clearly. 
conclusion: 
 Our paper introduces MoCi, a novel method that models multimodal contextual interactions of entities and achieves notable improved link prediction. MoCi addresses the deficiency of limited modal interactions within individual entities in most existing methods. Specifically, our proposed triplets-prompt modality contrastive pretraining strategy aligns modality semantics in advance. Moreover, explicit inter-entity modality interactions are effectively modeled by the proposed modality circular fusion method. MoCi represents the first endeavor to model multimodal contextual interactions from an inter-entity perspective. SOTA experimental results on seven datasets validate MoCi's efficacy in modeling inter-entity modality interactions for improving link prediction performance.  
: 
 For example, a textual description 𝑡 1 and visual image 𝑣 1 form the multimodal context of entity 𝑒 1 , denoted as 𝐶 𝑒 𝑖 = {𝑡 𝑖 , 𝑣 𝑖 }.Multimodal contexts capture rich information about entities and are leveraged to improve link prediction [29] .In this context, several models have been proposed to utilize multimodal information for link prediction. Among these, some studies have focused on projecting modality-specific information into a unified embedding space and incorporating it within entity embeddings [18, 26, 37] , as depicted in entity-level modality aggregation models (Figure 2 ). However, this approach might lead to the loss of unique information specific to each modality. In response, as depicted in Figure 2 , modality split and ensemble models have been proposed to address this limitation. These models emphasize the independent learning of distinct modality-specific features followed by their integration during the inference stage [34, 39] . Nonetheless, the modality interactions in these methods are typically confined within individual entity (entity-specific modality fusion). Thus neglecting the rich semantic information that can be derived from interactions between multimodal contexts of different entities, which could potentially contribute to the evaluation of triplets.Figure 1 provides illustrations of multimodal contextual interactions of entities. Take the scenario in Figure 1 as an example, where the triplet 𝑡𝑝 1 (Dustin_Hoffman, acted_in, Rain_Man) is a valid triplet, while 𝑡𝑝 2 (Dustin_Hoffman, acted_in, The_Lion_King) is an invalid triplet. The entities involved carry multimodal contexts, including textual and visual information, represented as 𝐶 ℎ𝑜 𝑓 𝑓 𝑚𝑎𝑛 = {𝑡 ℎ𝑜 𝑓 𝑓 𝑚𝑎𝑛 , 𝑣 ℎ𝑜 𝑓 𝑓 𝑚𝑎𝑛 }. In contexts where 𝑡𝑝 1 is valid and 𝑡𝑝 2 is not, the textual content 𝑡 ℎ𝑜 𝑓 𝑓 𝑚𝑎𝑛 ("Hoffman excels in antiheroes and vulnerable characters") exhibits semantic incompatibility with the visual information 𝑣 𝑇 ℎ𝑒_𝐿𝑖𝑜𝑛_𝐾𝑖𝑛𝑔 to a greater extent than with 𝑣 𝑅𝑎𝑖𝑛_𝑀𝑎𝑛 . Conversely, the semantic compatibility between 𝑡 ℎ𝑜 𝑓 𝑓 𝑚𝑎𝑛 and 𝑣 𝑅𝑎𝑖𝑛_𝑀𝑎𝑛 surpasses that between 𝑡 ℎ𝑜 𝑓 𝑓 𝑚𝑎𝑛 and 𝑣 𝑇 ℎ𝑒_𝐿𝑖𝑜𝑛_𝐾𝑖𝑛𝑔 . A similar pattern of semantic compatibility and incompatibility is observable in the visual context 𝑣 ℎ𝑜 𝑓 𝑓 𝑚𝑎𝑛 with respect to the textual descriptions of 𝑡 𝑅𝑎𝑖𝑛_𝑀𝑎𝑛 and 𝑡 𝑇 ℎ𝑒_𝐿𝑖𝑜𝑛_𝐾𝑖𝑛𝑔 , which are carried by the validity or invalidity of triplets. This implies that the semantic among different modalities of entities is consistent with the validity or invalidity of the triplets formed by these entities.Inspired by this, we propose to extract and learn semantic critical from the multimodal contexts of different entities (inter-entity modality interaction) for link prediction, rather than focusing solely on intra-entity modality interactions. To this end, we introduce a Modality Circular contrastive and fusion approach (MoCi). Specifically, a triplets-prompt contrastive pre-training strategy is introduced to align semantics across modalities. Subsequently, we employ a multilinear transformation strategy for modality circular fusion inter entity. Finally, relational context-aware predictions are integrated across modalities. To the best of our knowledge, MoCi represents a pioneering framework that shifts modality fusion in link prediction to the inter-entity perspective, thus significantly advancing accuracy in link prediction. Our contributions are as follows:• MoCi is the first work to advance multimodal fusion in the link prediction task toward an inter-entity perspective.  
 
 Appendix:  


Decision: The paper is mainly for multimodal link prediction, highlighting its importance and innovative approach, particularly through the proposed MoCi framework, which significantly enhances model performance by conducting inter-entity modality interactions. The extensive experiments and comparisons across seven datasets demonstrate the method's effectiveness. There were discrepancies in baseline results and a lack of explanations for these differences. Reviewers also noted issues with the figures, which were not intuitive and required clearer presentation. The complexity of the approach and potential computational inefficiencies were mentioned, along with the need for a more thorough discussion on interpretability, resource requirements, and robustness to variations in hyperparameters and dataset characteristics. Despite these limitations, the innovative nature of the method and its demonstrated performance improvements suggest that this paper should be accepted, provided the authors address the highlighted concerns.

Reviews: review: paper_summary: paper_summary: This paper proposes a novel approach called ModalityCircular fusion (MoCi), which models inter-entity modality interactions by interweaving the multimodal contexts of entities.

Unlike most methods that directly fuse modalities, the authors design a triplets-prompt modality contrastive pre-training to align modality semantics beforehand. Additionally, they introduce a modality circular fusion model employing a simple yet efficient multilinear transformation strategy, allowing explicit inter-entity modality interactions and distinguishing it from methods that fuse modalities within individual entities.
strength_weakness: strength_weakness: 1. Multimodal link prediction is an important foundational task.
2. The experimental results in this paper are very good. Although I reviewed some baseline papers (such as IMF) and found differences between the reported results and the original ones, I hope the authors can provide an explanation. However, even considering the original performance, the results in this paper are still impressive.
limitations: limitations: 1. The provided code is unusable, with no executable code available. This undermines my confidence in the reproducibility of this paper.
2. The writing of this paper needs improvement, as the expression is not intuitive. For example, can it clearly restate the connection between the methods and the motivations of the paper?
3. The figures in this paper are not intuitive. There are many unclear parts in the motivation and method diagrams, especially in the motivation diagram, which is really hard to understand.
4. The reason I gave Borderline Accept is that the method presented in this paper is quite interesting. However, if other reviewers find significant issues in the method, I will lower my rating.
paper_summary: paper_summary: This paper proposes a method called MoCi for link prediction task, which aligns modality semantics through a triplets-prompt modality contrastive pre-training strategy and achieves modality interaction between entities through a modality circular fusion model. Extensive experiments on multiple datasets have demonstrated the effectiveness of this method.
strength_weakness: strength_weakness: 1 This paper innovatively conducts inter-entity modality interactions, significantly enhancing the model's performance.
2 Extensive experiments and ablation studies validate the effectiveness of the proposed MoCi framework, showing a significant performance improvement over existing methods.
limitations: limitations: 1 There are several terms in the paper that I find hard to understand, such as ” structural bias”,"semantic critical," "semantic compatibility," and "triplets-prompt." Is it appropriate to use expressions like "prompt"? The author should provide clear definitions for these terms.
2 How are valid triplets and invalid triplets determined? How is the difference in "semantic compatibility" reflected in the example of Figure 1? Without prior knowledge, it is impossible to determine the relationship between Dustin_Hoffman and Rain_Man and Lion_King.
3 The writing of the paper is confusing; 'bt' and the 'structural loss function' are mentioned in Section 4.2, but their definitions are provided in Section 4.4.
paper_summary: paper_summary: This paper focuses on the multimodal knowledge graph completion (MKGC) task, which aims to infer missing triplets based on the multimodal information of entities and relations from the knowledge graph. The paper emphasizes the importance of inter-entity interaction, i.e., interactions of different entities. To this end, this paper proposes a two-stage training method, MoCi, which first learns multimodal embeddings for entities and relations with contrastive pre-training, then uses modality circular fusion to model inter-entity modality interactions. The proposed MoCi is evaluated on seven MKGC datasets compared with various baselines.
strength_weakness: strength_weakness: 1. Knowledge graph completion is an important task, and the utilization of multimodal content is helpful for real-world applications.
2. The proposed modality circular fusion is somewhat innovative.
3. The proposed method is compared with extensive baselines across seven datasets.
limitations: limitations: 1. **The motivation shown in Figure 1 is somewhat confusing.** The authors mention that $t_{hoffman}$ is more semantically compatible with $v_{rain\ man}$ compared to  $v_{the\ lion\ king}$ (lines 115 to 133). However, this is not intuitive. It appears that $t_{hoffman}$ and $t_{rain\ man}$ are more compatible because the text contains keywords like "vulnerable" and "autistic savant." The authors should further clarify this point.
2. **The experimental results are not very convincing.** It appears that there are differences between some baselines and the results reported in the original papers, but the authors do not explain the specific reasons. For example, IMF has an MRR of 0.485 on DB15K [1], whereas the authors' result is 0.441. Furthermore, in Table 3, why are different encoders used for different datasets? Do other baseline models use the same encoders? The authors should clarify these points to avoid unfair comparisons.
3. **Lack of efficiency experiments and analysis.** The authors use a two-stage training method, which raises concerns about potentially long training costs. Additionally, there is no analysis of the time complexity for the proposed Modality Circular Fusion (MCF) module.
4. **Lack of analysis of parameter sensitivity.** How should the best hyperparameters be selected? This would help understand the stability of the proposed model.

More Questions:
1. In line 339, what does $r_r^p$ represent, and does each relation $r$ also have multimodal representations? The authors only explain entity features in Table 1 but do not introduce relation features.
2. Are the reported results from a single run or the average of multiple runs? If it is the latter, the authors should provide the standard deviation or p-value to indicate the significance.
3. Why are the baselines different across datasets? It seems that the comparisons with baselines are selective across different datasets.

Reference

[1] IMF: Interactive Multimodal Fusion Model for Link Prediction. WWW 2023.
paper_summary: paper_summary: This paper introduces a novel approach called Modality Circular fusion (MoCi) for link prediction in knowledge graphs (KGs) by modeling inter-entity modality interactions. The key contributions include a triplets-prompt modality contrastive pre-training to align modality semantics, a modality circular fusion model for explicit inter-entity modality interactions, and relational context-aware predictions. The method is evaluated on seven datasets and achieves state-of-the-art performance.
strength_weakness: strength_weakness: 1. MoCi presents a novel approach that focuses on inter-entity modality interactions, which is not extensively explored in existing methods.
Technical Approach: The use of multimodal contextual information and the proposed fusion strategy show technical sophistication.
2. The method is validated on seven datasets, demonstrating its effectiveness and superiority over existing approaches.
limitations: limitations: 1. The complexity of the proposed approach, especially the modality circular fusion model, could potentially limit its computational efficiency, especially when applied to large-scale KGs or in real-time applications.
2. While the paper mentions explainability in the context of the circular fusion operation, a more thorough discussion or analysis of the interpretability of the learned representations and fusion process could be beneficial.
3. The paper does not discuss the data and resource requirements for implementing the proposed approach, including the size and nature of the KGs, the computational resources needed for training and inference, and the availability of pre-trained models for initialization.
4. It would be valuable to see an analysis of the robustness of the proposed approach to variations in hyperparameters, dataset characteristics, and modalities, as well as a sensitivity analysis to key design choices in MoCi.
5. The paper assumes that the multimodal data is of high quality and properly aligned, which might not always be the case in real-world scenarios. Addressing data preprocessing and cleaning could be a limitation. 
6. The model is designed for three modalities (text, image, and structure), and it might be challenging to generalize it to other unseen modalities without additional adaptation or fine-tuning.