IBMEA: Exploring Variational Information Bottleneck for Multi-modal Entity Alignment

Authors: taoyu  su
jiawei  sheng
shicheng  wang
xinghua  zhang
hongbo  xu
tingwen  liu
benedict  arnold
american  benedict
arnold  politician imageof
american  william
j  duane
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-03 14:28:30

Abstract: Multi-modal entity alignment (MMEA) aims to identify equivalent entities between multi-modal knowledge graphs (MMKGs), where entities can be associated with related images.
Most existing studies rely heavily on the automatically learned multi-modal fusion modules, which may allow redundant information such as misleading clues in the generated entity representations, impeding the feature consistency of equivalent entities.
To this end, we propose a variational framework for MMEA via information bottleneck, termed as IBMEA, by emphasizing alignment-relevant information while suppressing alignment-irrelevant information in entity representations.
Specifically, we first develop multi-modal variational encoders that represent modal-specific features as probability distributions.
Then, we propose four modal-specific information bottleneck regularizers to limit the misleading clues in the modal-specific entity representations. 
Finally, we propose a modal-hybrid information contrastive regularizer to integrate modal-specific representations and ensure the similarity of equivalent entities between MMKGs to achieve MMEA. 
We conduct extensive experiments on 2 cross-KG and 3 bilingual MMEA datasets.
Experimental results demonstrate that our model consistently outperforms previous state-of-the-art methods, and also shows promising and robust performance especially in the low-resource and high-noise data scenarios.

Main paper: 
 introduction: 
 Recent years have witnessed the booming of multi-modal knowledge graphs (MMKGs), which extend traditional knowledge graphs (KGs) by introducing multi-modal data, such as the relevant images of entities, to provide physical world meanings to the symbols of KGs. Along this line, various downstream applications can be supported, such as visual question answering [52] , recommendation systems [9, 48] and other applications [20, 50, 53] . However, MMKGs are usually constructed from separate multi-modal resources for different purposes, often suffering from the issue of incompleteness and limited coverage [24] . Therefore, the task of multi-modal entity alignment (MMEA) has been proposed, which aims to identify equivalent entities between two MMKGs, by accessing the multi-modal information of entities, such as their structures, relations, attributes, and images. In this way, one MMKG can retrieve and acquire useful knowledge from other MMKGs.To achieve MMEA, a prominent challenge is how to exploit the feature consistency of equivalent entities between MMKGs from their diverse and abundant multi-modal information. Pioneer methods such as PoE [24] concatenate all modality features to generate holistic multi-modal entity representations, but neglect the diverse importance of different modality features. Subsequently, effective methods [3, 13, 22, 23] attempt sophisticated multi-modal fusion modules to focus more on the important modality features, benefiting to emphasize consistent features for equivalent entities. For example, EVA [23] learns global attention weights for different modalities, and MEAformer [3] further develops dynamic modality weights specifically for different entities. However, these methods rely heavily on the automatically learned fusion module, which can be hampered by the alignment-irrelevant information such as misleading clues 1 in the entities' multi-modal information, especially when there are only low-resource or high-noise training data.In contrast, we argue that it is crucial to consider the misleading clues in modalities for MMEA. As shown in Figure 1 , given Entity_1 in MMKG-1, we are going to search the candidate entities (e.g., Entity_2, Entity_3, and so on) from MMKG-2, and identify Entity_2 as the true equivalent entity. However, we accidentally find that the image of Entity_1 has higher similarity 2 with Entity_3 due to the similar background color. Even though, we can still identify Entity_1 and Entity_2 as the equivalent entities by recognizing the details of human faces. Actually, in this case, the background color reflects the misleading clue that is alignment-irrelevant for MMEA. Existing methods usually struggle 3 with this case since the multimodal fusion modules may hardly discern the alignment-relevant information from the entire image information. This inspires us to emphasize the alignment-relevant information explicitly, and simultaneously surpass the alignment-irrelevant information in modalities, to relieve the impact of misleading clues.For this purpose, we explore information bottleneck (IB) [35, 36] as a potential solution. In general, the IB principle aims to learn an ideal representation that makes a trade-off between the fully data descriptive information and the task predictive information [36] . 1 It is also usually called shortcuts in researches [14] . 2 Note that we select examples from real datasets, and obtain image features from ResNet-152 to calculate cosine similarity. For details, please see Sec. 4.4.2. 3 For details, we show empirical results in Figure. 5, 6, and 7. For the MMEA task, we expect the multi-modal encoders of different modalities to extract alignment-relevant features as the task predictive information, while appropriately ignoring certain alignmentirrelevant information in the data descriptive information. In this way, the model is required to focus more on the necessary information for MMEA from the contexts of entity pairs between MMKGs, thus limiting the redundant information in entity representations and improving the robustness to misleading clues.Following the above idea, we propose a novel variational framework for Multi-modal Entity Alignment via Information Bottleneck, termed as IBMEA. Specifically, we first devise multi-modal variational encoders to capture the multi-modal features of each entity, including its graph structures, images, relations and attributes, and then represent them with probability distributions. Afterward, we propose two kinds of multi-modal information regularizers: 1) the modal-specific information bottleneck regularizer, which takes each modal-specific representation as input, surpassing alignmentirrelevant information and emphasizing alignment-relevant information, 2) the modal-hybrid information contrastive regularizer, which adaptively fuses all modal-specific representations to generate modal-hybrid representations, and further enhances entity similarity between entity pairs to achieve MMEA. Our major contributions can be summarized as follows:• We introduce a novel perspective for MMEA with IB, and propose a variational framework as IBMEA. To our knowledge, we are the first to explore IB to alleviate misleading clues in MMEA. • We propose two kinds of regularizers to refine the modal-specific and modal-hybrid features, which surpass alignment-irrelevant information and emphasize alignment-relevant information, improving the robustness with misleading clues.• Experiments indicate that our model outperforms the comparison state-of-the-art methods on 5 benchmarks, and obtains promising and robust results in low-resource and high-noise scenarios. 
preliminaries 2.1 task formulation: 
 Formally, the multi-modal knowledge graph (MMKG) can be defined as G = (E, R, A, V), where E, R, A, V are the sets of entities, relations, attributes and visual images, respectively. Therefore, the triples are defined as T ⊆ E × R × E, where each entity 𝑒 ∈ E can be linked to attributes and images. Building on previous research [3, 22, 23] , we mainly focus four modalities 𝑚 ∈ {𝑔, 𝑣, 𝑟, 𝑎} of entities, including graph structures 𝑔, visual images 𝑣, neighboring relations 𝑟 and attributes 𝑎.Based upon, multi-modal entity alignment (MMEA) aims to identify equivalent entities from two different MMKGs. Formally, given two MMKGs G (1) and G (2) with their relational triples and multi-modal attributes, the goal of MMEA task is to identify equivalent entity pairs {⟨𝑒 (1) , 𝑒 (2) ⟩|𝑒 (1) ∈ G (1) , 𝑒 (2) ∈ G (2) , 𝑒 (1) ≡ 𝑒 (2) }.During training, the model utilizes a set of pre-aligned entity pairs S as alignment seeds, while testing focuses on predicting equivalent entity pairs across MMKGs. In this paper, we aim to achieve MMEA by learning entity representations from both G (1) and G (2) , emphasizing alignment-relevant information while surpassing alignmentirrelevant information to bridge between MMKGs. We adopt the information bottleneck (IB) principle [35, 36] to learn general entity representations across MMKGs. The IB principle aims to find a maximally compressed representation 𝒁 of the input 𝑿 while preserving information about the target 𝒀 , which achieved by minimizing the objective function:EQUATIONwhere 𝐼 (•; •) denotes mutual information, and 𝛽 > 0 is a Lagrangian multiplier controlling the trade-off between compression and information preservation. Minimizing the first term penalizes the information between 𝒁 and 𝑿 , encouraging the latent variable 𝑍 to forget irrelevant information. Maximizing the second term encourages 𝒁 to be predictive of 𝒀 . Consequently, the IB principle enables 𝒁 to capture relevant factors for prediction while compressing irrelevant parts [6] . This aligns with the notion of 𝒁 acting as a minimal sufficient statistic of 𝑿 for predicting 𝒀 [5] .Although the IB principle is appealing, computing mutual information poses computational challenges. To overcome this, the VIB (Variational Information Bottleneck) [5] method proposes a variational approximation approach for the IB objective. It minimizes the following formulation:EQUATIONwhere 𝑝 𝜃 (𝒛|𝒙) is an estimate of posterior probability to generate 𝒛 from data 𝒙, 𝑟 (𝒛) is an estimate of the prior probability 𝑝 (𝒛) to describe 𝒛, and 𝑞 𝜙 (𝒚|𝒛) is a parametric approximation of 𝑝 (𝒚|𝒛) to predict target 𝒚. Intuitively, the encoder generates 𝑝 𝜃 (𝒛|𝒙) with parameter 𝜃 , and the decoder achieves 𝑞 𝜙 (𝒚|𝒛) with parameter 𝜙. Actually, Eq. ( 2) resembles a likelihood-prior trade-off, since the first term applies KL-divergence with uninformative prior, penalizing the too complex encoded representations, and the second term approximates the log-likelihood expectation, requiring the representations to achieve task prediction. 
methodology: 
 The overall framework of IBMEA is illustrated in Figure 2 . Our model mainly consists of two components: 1) Multi-modal Variational Encoder, which represents multi-modal information of entities from both MMKGs into random variables, and 2) Multi-modal Information Regularizer, which imposes IB regularization on each modal-specific representations, and facilitates MMEA with contrastive regularization upon modal-hybrid representations.To capture entity features from all modalities, including the graph structure 𝑔, visual image 𝑣, relation 𝑟 and attribute 𝑎. We devise the modal-specific variational encoder and generate the modal-specific feature variables as 𝒛 𝑚 with 𝑚 ∈ {𝑔, 𝑣, 𝑟, 𝑎}, respectively.To fully leverage the graph structures of MMKGs, we devise a variational graph encoder. Considering the neighbors of an entity can have different importance for entity alignment, we adopt (two attention heads and two layers) graph attention network (GAT) [40] to capture graph structures.To better depict the information of the graph structural feature, we represent it with a variable 𝒛 𝑔 of the probability distribution. Practically, we assume the variable as Gaussian distributions as widely used in variational studies [5, 16, 44] , and represent 𝒛 𝑔 with mean vector 𝝁 𝑔 and variance vector 𝝈 𝑔 , derived by separate GATs with different learnable parameters [16] :EQUATIONwhere 𝒙 𝑔 ∈ R 𝑑 𝑔 is the randomly initialized node features, 𝑨 denotes the adjacent matrix of the given MMKG. To alleviate the negative impacts of structural heterogeneity, we encode both MMKGs with shared GATs to project entities into the same embedding space [18] . The learnable parameters can be summarized as 𝜃 𝑔 ≜ {𝜃 𝑔,𝜇 , 𝜃 𝑔,𝜎 }.To highlight the information of relations, attributes, and images, we devise separate fully connected layers as multi-modal encoders to learn interim representations for each modality 𝑚 ′ ∈ {𝑣, 𝑎, 𝑟 } as follows:EQUATIONwhere 𝑾 𝑚 ′ ∈ R 𝑑 𝑚 ′ ×𝑑 and 𝛿 is the ReLU activation. Here, 𝒙 𝑚 ′ ∈ R 𝑑 𝑚 ′ denotes the initialized feature for 𝑚 ′ -modality. For images, we utilize pre-trained visual encoders to obtain effective image features. For entities without images, we average the images of their neighbors as the initial feature [19] . For attributes and relations, the bag-of-attributes and bag-of-relations features [49] are employed.Based on these interim representations, we employ multi-layer perceptrons (MLPs) to derive modal-specific feature variables. Assuming these variables follow a Gaussian distribution and employ separate MLPs to learn mean vector 𝝁 𝑚 ′ and variance vector 𝝈 𝑚 ′ :EQUATIONwhere the learnable parameters for 𝑚 ′ -modality can be summarized as𝜃 𝑚 ′ ≜ {𝜃 𝑚 ′ ,𝜇 , 𝜃 𝑚 ′ ,𝜎 , 𝑾 𝑚 ′ } with 𝑚 ′ ∈ {𝑣, 𝑎, 𝑟 }.Since the modalspecific variables are intractable for neural networks, we leverage the reparameterization trick [16] to sample deterministic representations from probability distributions. For each modality 𝑚 ∈ {𝑔, 𝑣, 𝑎, 𝑟 }, the sampled representation 𝒛 𝑚 is obtained as follows:EQUATIONwhere 𝝁 𝑚 and 𝝈 𝑚 are the corresponding mean and standard deviation of 𝑚-modality, respectively. 𝝐 is a standard Gaussian noise, and ⊙ denotes the element-wise production. In this way, we obtain the deterministic modal-specific representations for entities, which can be further integrated to acquire entire (modal-hybrid) entity representations to make predictions for MMEA.In this section, we introduce two kinds of regularizers from mutual information perspective, which refine the modal-specific representations with limited alignment-irrelevant information, and derive better modal-hybrid representations for MMEA.As claimed in Sec. 1, there may exist alignment-irrelevant misleading clues in distinct modalities. For each modality 𝑚 ∈ {𝑔, 𝑣, 𝑎, 𝑟 }, we expect the learned modal-specific feature variables 𝒛 𝑚 to contain less alignment-irrelevant information while retaining more alignmentrelevant information between MMKGs. Following this idea, we propose the information bottleneck regularizer L 𝑚 for 𝑚-modality feature variables of the paired entities from the two MMKGs:EQUATION𝑚 ; 𝑿EQUATION𝑚 ; 𝑿𝑚 )EQUATION𝑚 , 𝒁EQUATIONwhere 𝑿𝑚 and 𝑿(2)𝑚 denotes the original 𝑚-modality features from G (1) and G (2) , respectively. 𝒁 𝑚 are obtained by Eq. ( 6) over the two MMKGs. 𝒀 denotes the supervised information, indicating whether the paired entities are equivalent. Note that the minimality term penalizes the information from its own MMKG, while the alignment term encourages the representations to be capable of entity alignment. In this way, the redundant information is surpassed by minimality term, alleviating alignment-irrelevant misleading clues in the modal-specific representations. On the other hand, the alignment-relevant information is simultaneously emphasized, since we require the modal to make prediction by the alignment term with tailored information. For tractable objective function, please refer to Sec. 3.3.1.To refine modal-specific representations of all modalities, the overall IB regularizer L 𝑠𝑝𝑒𝑐𝑖 𝑓 𝑖𝑐 is defined as follows:EQUATIONIn this way, we encourage each modality of information to remain consistency between MMKGs, benefiting to entire representations.Regularizer. To fully leverage information of all modalities for MMEA, we propose a modal-hybrid information contrastive regularizer. We first generate modal-hybrid feature variables by integrating all the modal-specific feature variables. Specifically, for each entity, we measure the distinct importance of its modality information with attention mechanism, and employ the attention weights to integrate modal-specific feature variables (sampled from Eq. ( 6)) as follows:𝑠 𝑚 = 𝒒 𝑇 𝑜 tanh(𝑾 𝑚 𝒛 𝑚 + 𝑏 𝑚 ), 𝛼 𝑚 = exp(𝑠 𝑚 ) 𝑘 ∈ {𝑔,𝑣,𝑎,𝑟 } exp(𝑠 𝑘 ),EQUATIONwhere 𝛼 𝑚 is the attention weight for modality 𝑚, taking the different nature of entities into consideration. In addition, 𝒒 𝑜 ∈ R 𝑑 , 𝑾 𝑚 ∈ R 𝑑 ×𝑑 𝑚 and 𝑏 𝑚 ∈ R 𝑑 are learnable parameters. In this way, we obtain modal-hybrid feature variables considering the distinct modality importance of the entity and leverage the IB-refined modal-specific feature variables.Based on the derived modal-hybrid feature variable, we aim to measure the mutual information between paired entities from the two MMKGs. Our main intuition is that, given an entity in the one MMKG, the equivalent entity in the other MMKG would have higher mutual information than other candidate entities. Following this intuition, we propose the modal-hybrid information contrastive regularizer L ℎ𝑦𝑏𝑟𝑖𝑑 as follows:EQUATION)In this way, we measure the consistency between paired entities from the two MMKGs, which fully leverages all the IB-refined modal-specific features to constitute the entire entity representations. For tractable objective function, please refer to Sec. 3.3.2.Theoretically, direct optimization of mutual information can be intractable considering the intractable integrals [6, 29] . Therefore, we derive tractable objectives with variational approximation [38] .Objective. To optimize the information bottleneck regularizer in Eq. ( 7), we derive the variational lower bound for the minimality and alignment term.For the minimality term, we measure it by Kullback-Leibler (KL) divergence [4] with variational approximation posterior distributions. Specifically, recall that we are given the original feature 𝒙 𝑚 of 𝑚-modality from G (𝑚 ∈ {𝑔, 𝑣, 𝑟, 𝑎}, and we omit the subscript of different MMKGs), and the feature variable 𝒛 𝑚 can be approximated by the 𝑚-th modality variational encoder. We can derive the approximation upper bound [29] of the minimality term, and minimize it as the tractable objective function:EQUATIONHere, to approximate true posterior distributions 𝑝 (𝒛 𝑚 |𝒙 𝑚 ), we adopt the 𝑚-modality variational encoder parameterized by 𝜃 (omit subscript 𝑚 of 𝜃 𝑚 ) to generate approximated posterior distribution as 𝑝 𝜃 (𝒛 𝑚 |𝒙 𝑚 ). Besides, following existing variational methods [5, 16, 44] , we also assume the prior distribution 𝑟 (𝒛 𝑚 ) as a standard Gaussian distribution N (0, diag(𝑰 )) for convenience. Such a procedure holds for both MMKG G (1) and G (2) . In this way, we compress redundant information in the 𝑚-modality representation.For the alignment term, we also adopt variational approximation posterior distribution to render. The goal of the alignment term is to make prediction with the learned feature variables. Therefore, we rewrite the alignment term with likelihood of the entity alignment task, which is the variational lower bound for maximization as the tractable objective function:EQUATIONEQUATIONEQUATION𝑚 )], (12) where the variable 𝒚 denotes whether the entity pair is equivalent, and 𝑞 𝜙 (𝒚|𝒛 (1)EQUATION𝑚 ) is the entity alignment decoder to make prediction. Generally, the likelihood can be achieved by binary cross-entropy, and here we use InfoNCE [39] to better consider all the candidate entities in ranking, which can be formed as:EQUATIONEQUATION𝑚 ) ∈ S log exp(cos(𝒛EQUATIONEQUATIONEQUATIONEQUATION)where S is the seed alignments (i.e., pre-aligned entity pairs), 𝜏 is the temperature factor. ẑ (2) 𝑚 is the negative entity obtained by randomly replacing the true entity of seed alignments.To optimize the contrastive term, we follow the idea of InfoMax [41] by measuring the mutual information with neural networks. Specifically, we use another discriminator D to measure the consistency between entities, and the lower bound of the contrastive term can be derived as:𝐼 (𝒁 (1) 𝑜 ; 𝒁 (2) 𝑜 ) = E 𝑝 (𝒛 (1)𝑜 |𝒙 (1) )𝑝 (𝒛 (2) 𝑜 |𝒙 (2) )[log D(𝒛EQUATION𝑜 ,𝒛𝑜 ) ∈S𝑜 , 𝒛EQUATION𝑜 ,𝒛EQUATION𝑜 , 𝒛𝑜 ) ), (14) where 𝒛𝑜 and 𝒛𝑜 are modal-hybrid variables for G (1) and G (2) from Eq. ( 9) respectively, which encode all the original modality features (𝒙 ≜ {𝑥 𝑔 , 𝑥 𝑣 , 𝑥 𝑎 , 𝑥 𝑟 }) in the corresponding MMKG. Here we can achieve the discriminator D with inner production followed by sigmoid. Inspired by Boudiaf et al. [8] , we adopt multi-similarity loss [43] to further consider the impact of different entities in contrastive learning. In this way, we encourage the modal-hybrid representations of the equivalent entities to be relevant and thereupon achieve the MMEA task with all modality information.Since we have imposed constraints on both the modal-specific and modal-hybrid representations, the overall objective function is defined in a joint paradigm as follows:EQUATIONIn summary, the proposed IB regularizers enable us to suppress alignment-irrelevant information and emphasize the alignmentrelevant information to fulfill the MMEA task. 
experiments: 
 In this section, we conduct extensive experiments to evaluate the effectiveness. More analyses are detailed in Appendix4 .In this study, we evaluate our proposed IBMEA on five multi-modal EA datasets, categorized into two types. (1) Cross-KG [24] datasets, comprising FB15K-DB15K and FB15K-YAGO15K, with 128,486 and 11,199 labeled pre-aligned entity pairs, respectively. (2) Bilingual datasets also named DBP15K [23, 32] , which are including DBP15K ZH-EN , DBP15K JA-EN and DBP15K FR-EN from the multilingual versions of DBpedia, each with approximately 400K triples and 15K pre-aligned entity pairs. EVA [23] provided images of entities for the DBP15K dataset. Following previous works [3, 22, 23] , we utilize 20%, 50%, 80% of true entity pairs as alignment seeds for training on cross-KG datasets, and 30% for bilingual datasets. We use Hits@1 (H@1), Hits@10 (H@10), and Mean Reciprocal Rank (MRR) as evaluation metrics. Hits@N denotes the proportion of correct entities in the top N ranks, while MRR is the average reciprocal rank. For both metrics, higher values indicate better alignment results.To verify the effectiveness of IBMEA, we select several typical and competitive methods as baselines. We manifest these methods into two groups: Traditional Entity Alignment methods. We choose 6 prominent EA methods proposed in recent years, which achieve entity alignment based on the graph structures and do not introduce multi-modal information, including TransE [7] , IPTransE [54] , GCN-align [45] , KECG [18] , BootEA [33] , and NAEA [55] . MMEA methods. We further collect 10 state-ofthe-art MMEA methods, which incorporate entity images as input features to enrich entity representations, including POE [24] , Chen Methods FB15K-DB15K (20%) FB15K-DB15K (50%) FB15K-DB15K (80%) FB15K-YAGO15K (20%) FB15K-YAGO15K (50%) FB15K-YAGO15K (80%)H@1 H@10 MRR H@1 H@10 MRR H@1 H@10 MRR H@1 H@10 MRR H@1 H@10 MRR H@1 H@10 MRR TransE [7] . et al. [10] , HMEA [15] , EVA [23] , ACK-MMEA [19] , MSNEA [11] , PSNEA [28] , UMAEA [13] , MCLEA [22] , and MEAformer [3] . For more details, please refer to Sec. 5. Among the methods, MCLEA and MEAformer are typical competitive methods.In our experiments, the GAT has two layers with a hidden size of 𝑑 𝑔 = 300. For visual embeddings, following [3, 22, 23] , we leverage a pre-trained VGG-16 model [31] for cross-KG datasets with 𝑑 𝑣 = 4096 and ResNet-152 for bilingual datasets with 𝑑 𝑣 = 2048 to obtain initial features from the penultimate layer. Following [3, 22] , the Bag-of-Words method is chosen to encode both attributes (𝒙 𝑎 ) and relations (𝒙 𝑟 ) as fixedlength vectors, where 𝑑 𝑎 and 𝑑 𝑟 are both 1000. The graph embedding output is 300, and other modality embeddings are 100. Training is conducted over 1000 epochs with a batch size of 7,500, using AdamW optimizer [26] with a learning rate of 6e-3 and a weight decay of 1e-2. Hyper-parameters 𝛽 𝑔 , 𝛽 𝑣 , 𝛽 𝑎 , 𝛽 𝑟 in Eq. ( 7) are tune in [1e-4, 1e-3, 1e-2, 1e-1], yielding optimal results at 1e-3, 1e-2, 1e-2, 1e-2, respectively. Consistent with prior studies [3, 22, 23] , we adopt an iterative training strategy to overcome the lack of training data and exclude entity names for fair comparison. Our best hyper-parameters are tuned by grid search according to the prediction accuracy of MMEA, detailed in Appendix.The overall average results on cross-KG and bilingual datasets are displayed in Table 1 and Table 2 , respectively. From the tables, we have several observations: 1) Our proposed IBMEA model outperforms all baseline models on 5 benchmarks with different data settings, including two cross-KG datasets and three bilingual datasets, excelling in all key metrics (H@1, H@10, and MRR), which demonstrates the model's generality. Specifically, under 50% and 80% alignment seed settings on 2 cross-KG datasets, our model achieved an average increase of 4.2% and 3.0% in H@1 scores, and 3.5% and 2.1% in MRR scores, respectively. Moreover, our model still exceeds those high-performing baselines and increases the current SOTA Hits@1 scores from .847/.842/.845 to .859/.856/.864 on ZH-EN/JA-EN/FR-EN datasets with the DBP15K, respectively. 2) Our model achieves better results in relatively low-resource data scenario. Compared to the runner-up method results, our model achieves an average increase of 4.4% in H@1 and 4.0% in MRR on cross-KG datasets with a 20% alignment seed setting. It demonstrates that using the IB principle, the model can better grasp alignment-relevant information, alleviating overfitting on pieces of predictive clues with limited seed alignments.3) The MMEA baseline model generally outperforms the EA baseline. Remarkably, our model utilizing 20% of the seeds surpasses the H@1 performance of the EA baseline at 80% in cross-KG datasets. It demonstrates that introducing multi-modal information can enrich entity information, and significantly help entity alignment with limited seed alignments. All the results demonstrate the effectiveness of our proposed IBMEA, and we will further examine our model on hard data scenarios to show the results in Sec. 4.4.To evaluate the unity of all regularizers, we conduct ablation study on FB15K-DB15K dataset. From the Table 3 , we can observe that:1) The removal of any modal-specific IB regularizer consistently leads to significant reductions across all metrics, thereby validating its efficacy for modal-specific IB regularizers. 2) As the proportion of alignment seeds increases, the negative impact of removing the Table 3 : Ablation study on IB regularizers. G-IB, V-IB, A-IB, and R-IB are modal-specific IB regularizers (Graph, Visual, Attribute, and Relation) for short. Hybrid-IB refers to using an IB regularizer on the representations after fusion.Model FB15K-DB15K (20%) FB15K-DB15K (50%) FB15K-DB15K (80%)H@1 H@10 MRR H@1 H@10 MRR H@1 H@10 MRR To further evaluate the impact of different modalities, we report the results by deleting different modalities on FB15K-DB15K dataset, shown in Figure 3 . We notice that: 1) Removing each modality will reduce the final result, proving each modality's importance. 2) Compared with the high proportion of seeds (50% and 80% ratio), the effect of different modalities information on the overall performance is more obvious in the low proportion of seeds. This indicates that our model can extract crucial information of each modality, thus obtaining better results in the case of the low proportion of seeds. To further explore the performance with few training data, where the alignment seed ratio ranges from 5% to 30%. Specifically, we select MCLEA and MEAformer as baselines, and the results are depicted in Figure 4 . We can observe that as the proportion of alignment seeds decreases, the performance of all models would also decrease in terms of metrics. However, it is obvious that our IBMEA continuously outperforms MCLEA and MEAformer, indicating the effectiveness of our proposed method. Moreover, it is worth noting that the gap between them is much more significant when the seed alignments are extremely few (5%), which confirms the superiority of our proposed method to alleviate the overfitting of shortcut misleading clues in low-resource scenarios.4.4.2 Performance on samples with low-similarity image. To examine the effectiveness of our models on hard samples, we select samples with low-similarity image in FB15K-DB15K dataset (20% seed alignments) and examine the performance of the three MMEA models on these samples. The similarity between the images is calculated by the cosine similarity of the features extracted from the pre-trained visual encoder. Considering alignment entity pairs with low-similarity image, they tend to contain more task-irrelevant misleading information. As shown in Figure 5 , our model achieves the best results on both the H@1 and MRR metrics with different entity image similarity settings. We believe that our model can effectively leverage IB principle in training, which encourages the model to focus more on the task-relevant image information for prediction rather than the overall redundant image information.To explore the efficacy of entity alignment in noisy data scenarios, we utilize FB15K-DB15K dataset (20% seed alignments) and add artificial noise to the images for experiments. Specifically, we first obtain the entity's image embedding from the pre-trained visual encoder, and then impose random dropout to them with a dropout rate to control the noise rate. As depicted in Figure 6 , results indicate that all models reflect a performance decline with increasing the image noise rate. However, even under a higher noise rate, our model maintains relatively high H@1 and MRR metrics. Considering the higher noise rate tends to have more misleading clues, the results demonstrate the effectiveness of our model with IB regularizers, which successfully preserves alignment-relevant information while suppressing alignment-irrelevant information, thereby sustaining superior alignment performance in noisy environments.To detail the model prediction, we present typical cases of aligned entities from DBP15K ZH-EN with relatively low-similarity figures in Figure 7 . Case 1 shows characters with different clothes but similar facial features. Case 2 displays maps with different background colors yet identical contour features. In Case 3, a statue and a black-and-white photo of the same person exhibit similar facial traits, providing crucial information for entity alignment. The three cases reflect misleading clues in images. As in the predictions, it was found that MCLEA and MEAformer predict the true entity with a lower ranking. Note that in Case 3, the MEAformer model predicts worse, likely due to being misled by differences in entity forms. In contrast, our model accurately predicted in all cases, thus demonstrating its superior ability to handle misleading information for the MMEA task. 
related work 5.1 entity alignment: 
 Entity Alignment (EA) aims to identify equivalent entities across different knowledge graphs (KGs) to facilitate knowledge fusion. The majority of existing EA methods focus on traditional knowledge graphs can be classified into two categories: 1) Structure-based techniques focus on employing knowledge embedding [1, 7, 12, 33, 54] to capture the entity structure information from relational triples, or utilize graph-based models [18, 27, 34, 45] for neighborhood entity feature aggregation [17, 21, 30, 40] . 2) Side information-based methods [2, 25, 32, 37, 46] integrate side information (e.g., entity name, entity attribute, relation predicates) to learn more informative entity representation. As previous studies [25, 51] have shown, the structure-based techniques assume that aligned entities should share similar neighborhoods, the side information-based methods regard equivalent entities usually contain similar side information. Nevertheless, these methods mainly solely on structural information and utilize textual side information, making them incapable of handling the visual data of entities in actual scenes. Due to the increasing popularity of multi-modal knowledge graphs, how to incorporate visual modalities in EA, i.e. multi-modal entity alignment (MMEA), has attracted research attention. The pioneer method PoE [24] defines overall probability distribution as the product of all uni-modal experts. Chen et al. [10] design a multimodal fusion module for integrating different modal embeddings. HMEA [15] combines the structure and visual representations in hyperbolic space. MSNEA [11] and XGEA [47] integrate visual features to guide relational and attribute learning, with the former using a translation-based KG embedding method and the latter using a graph neural network (GNN) approach. ACK-MMEA [19] proposes a method for uniformizing multi-modal attributes, while UMAEA [13] introduces multi-scale modality hybrid and circularly missing modality imagination. PSNEA [28] and PCMEA [42] dynamically generate pseudo-label data to improve alignment performance. EVA [23] allows the alignment model to obtain the importance of different modality weights to fuse multi-modal information from KGs into a joint embedding. MCLEA [22] follows the EVA fusion method and then obtains informative entity representations based on contrastive learning. MEAformer [3] develops UMAEA and dynamically learns modality weights for each entity via the transformer-based attention fusion method. However, the above existing methods rarely explicitly address the misleading alignmentirrelevant information in each modality, resulting in incomplete utilization of the alignment-relevant information. 
conclusion: 
 In this paper, we explore MMEA to identify equivalent entities between MMKGs. To address alignment-irrelevant misleading clues in modalities, we propose a novel MMEA framework termed as IBMEA, which aims to emphasize alignment-relevant information and simultaneously suppress alignment-irrelevant information in modalities. Particularly, we devise multi-modal variational encoders to represent multi-modal information with probability distributions and propose information bottleneck regularizers and an information contrastive regularizer. Experimental results indicate that our model outperforms previous state-of-the-art methods, and shows promising capability for misleading information alleviation, especially in the low-resource and high-noise data scenarios. 
 
 Appendix: Our Appendix and code are available at https://github.com/sutaoyu/IBMEA. 


Decision: I have went through the entire process, including the submission, responses and discussion phases. The reviewers raised up some concerns to the authors, while the authors made the clarifications. However, the reviewers are NOT posing the final justifications and comments/discussions based on the authors' responses. I encouraged the authors to take the comments into considerations for their final version, such as clarifying some misunderstandings. I hence recommend the acceptance as a poster.

Reviews: review: paper_summary: paper_summary: The research paper discusses Multi-modal Entity Alignment (MMEA) in the context of multi-modal knowledge graphs (MMKGs), where entities are linked to related images. Traditional methods struggle with irrelevant and misleading information, particularly in data-poor or noisy conditions. To address this, the authors propose a novel framework called IBMEA, which uses the Information Bottleneck (IB) principle to enhance entity alignment by focusing on relevant information and filtering out irrelevant data. The framework employs multi-modal variational encoders and introduces two types of regularizers: modal-specific information bottleneck regularizers and a modal-hybrid information contrastive regularizer. These components work together to refine and integrate entity representations, improving alignment accuracy. Extensive experiments demonstrate that IBMEA surpasses existing methods, showing strong performance even in challenging scenarios.
strength_weakness: strength_weakness: 1.The paper introduces a novel approach to Multi-modal Entity Alignment (MMEA) by leveraging the Information Bottleneck (IB) principle. This is the first known application of IB to alleviate the impact of misleading clues in MMEA, providing a fresh perspective on enhancing entity alignment through explicit emphasis on relevant information and suppression of irrelevant data.

2. The proposed IBMEA framework has significant applications in the field of knowledge graph alignment, particularly in scenarios involving multi-modal data. Its ability to handle low-resource and high-noise environments makes it a valuable tool for real-world applications where data quality and quantity can be challenging.

3. The paper provides extensive empirical evaluation on five benchmark datasets, including two cross-KG and three bilingual MMEA datasets. The experiments results demonstrated that IBMEA consistently outperforms state-of-the-art methods.
limitations: limitations: The main issues of this article lie in the following two aspects:

**1. Unclear Relationship Between Information Bottleneck Theory and Multi-modal Alignment**

1) The connection between information bottleneck (IB) theory and multi-modal entity alignment (MMEA) is not clearly established. For instance, Figure 1 does not effectively illustrate the application of IB in the context of MMEA. Actually, the image similarity can be enhanced by focusing on specific modal information, such as grayscale or texture images, which resembles the concept of background removal. However, this does not convincingly demonstrate the role of IB.

2) The paper introduces IB to address the representation and identification of alignment-relevant and irrelevant features, a common issue in multi-modal data fusion and entity alignment. However, the physical meaning of IB in the context of MMEA remains ambiguous. The paper should reference prior works, such as [33], to better define the problem and clarify the relevance of IB. Additionally, the rationale for using IB to solve these issues, and the specific attributes of IB that make it suitable for feature representation in MMEA, need further explanation. This ambiguity undermines the paper's motivation and makes it appear as if a general theory is being applied to a specific application without sufficient justification.

3) In the methodology section, the necessity of IB for MMEA tasks is not adequately summarized. The paper does not explain why IB is essential, nor does it provide the theoretical background supporting its use. The Modal-specific Information Bottleneck Regularizer appears to merely apply mutual information, and its theoretical connection to IB needs to be further clarified.

**2. Inconsistencies in Thesis Arguments**

1) The paper claims to focus on multi-modal entity alignment, but the bilingual datasets used in the experiments do not appear to include multi-modal data such as images. The source and usage of image data are not provided or explained. This discrepancy needs to be addressed to maintain consistency in the paper's arguments.

2) While the paper emphasizes the theoretical application of IB in MMEA, the experimental section lacks theoretical analysis and proof of effectiveness regarding the IB principle. There are no experiments specifically measuring the impact of IB. The paper should include a detailed theoretical analysis and experimental verification of how the IB principle is effectively utilized, which would strengthen the overall argument and contribution of the study.


To sum up, the motivation of this paper is interesting, and the application of the information bottleneck principle is relatively novel. However, the two main points discussed above require further clarification. Additionally, more extensive effectiveness analysis is necessary, such as measurement experiments of information bottlenecks in MMEA tasks. In my opinion, the paper requires major revisions. I hope my comments will be helpful to the authors.
paper_summary: paper_summary: This paper proposes a variational information bottleneck for multi-modal entity alignment (IBMEA), suppressing redundant modality information between multi-modal knowledge graphs. It also proposes modal-specific information bottleneck regularizers to limit the misleading clues in refining modal-specific entity representations. Experimental results demonstrate that the model consistently outperforms previous state-of-the-art methods, and also shows promising and robust performance in low-resource and high-noise data scenarios.
strength_weakness: strength_weakness: 1. This paper designs a novel variational framework with information bottleneck (IB) principle, explores IB to alleviate misleading clues in MMEA. The experimental section well demonstrates the strategy's effectiveness.
2. It proposes 2 regularizers to refine the modal-specific and modal-hybrid features, improving the robustness with misleading clues.
3. The paper is well written and organized.
limitations: limitations: 1. I hope to see more experimental evidence demonstrating the effectiveness of this two kinds of regularizers.
2. It's lacking in novelty in utilizing IB to suppress redundant modality information.
paper_summary: paper_summary: This paper proposes a novel Variational Information Bottleneck Method (IBMEA) for Multimodal Entity Alignment (MMEA). MMEA aims to identify equivalent entities in different Multimodal Knowledge Graphs (MMKGs), which are typically associated with relevant images. Existing studies often fail to effectively suppress alignment-irrelevant information when fusing multimodal information, leading to subpar performance in low-resource or high-noise data scenarios. To address this issue, the paper suggests using the Information Bottleneck to emphasize alignment-relevant information and suppress irrelevant information, thereby generating entity representations. Specifically, the method includes designing a multimodal variational encoder to generate modality-specific entity representations and proposes four modality-specific information bottleneck regularization methods and a modality-mixed information contrastive regularization method.
strength_weakness: strength_weakness: 1. IBMEA is the first to apply the Information Bottleneck (IB) to multimodal entity alignment, providing a new perspective to address the problem of misleading information during alignment.
2. The paper introduces two types of regularization methods: modality-specific information bottleneck regularization and modality-mixed information contrastive regularization, which optimize modality-specific and mixed representations, respectively, enhancing the model's robustness against misleading information.
3. Extensive experiments on five benchmark datasets demonstrate that IBMEA surpasses existing state-of-the-art methods across various data settings, showing exceptional performance in low-resource and high-noise data scenarios.
limitations: limitations: 1. Complexity: The application of the Information Bottleneck principle to multimodal entity alignment by IBMEA is very novel and promising, warranting further exploration of how this approach can be applied to other multimodal tasks.
2. Interpretability: While the proposed method performs excellently in experiments, the internal mechanisms of the model and the specific contributions of different regularization terms may not be easily interpretable, requiring further research to enhance the model's transparency.