Tackling Uncertain Correspondences for Multi-Modal Entity Alignment
Authors: Liyi Chen
Ying Sun
Shengzhe Zhang
Yuyang Ye
Wei Wu
Hui Xiong
Venue: NeurIPS 2024 poster (2024)
Keywords: Entity alignment
knowledge graph
multi-modal learning
Primary area: natural_language_processing

Abstract: Recently, multi-modal entity alignment has emerged as a pivotal endeavor for the integration of Multi-Modal Knowledge Graphs (MMKGs) originating from diverse data sources. Existing works primarily focus on fully depicting entity features by designing various modality encoders or fusion approaches. However, uncertain correspondences between inter-modal or intra-modal cues, such as weak inter-modal associations, description diversity, and modality absence, still severely hinder the effective exploration of aligned entity similarities. To this end, in this paper, we propose a novel Tackling uncertain correspondences method for Multi-modal Entity Alignment (TMEA). Specifically, to handle diverse attribute knowledge descriptions, we design alignment-augmented abstract representation that incorporates the large language model and in-context learning into attribute alignment and filtering for generating and embedding the attribute abstract. In order to mitigate the influence of the modality absence, we propose to unify all modality features into a shared latent subspace and generate pseudo features via variational autoencoders according to existing modal features. Then, we develop an inter-modal commonality enhancement mechanism based on cross-attention with orthogonal constraints, to address weak semantic associations between modalities. Extensive experiments on two real-world datasets validate the effectiveness of TMEA with a clear improvement over competitive baselines.

Tackling Uncertain Correspondences for Multi-Modal Entity Alignment
Liyi Chen', Ying Sun', Shengzhe Zhang', Yuyang Ye Wei Wu', Hui Xiong University of Science and Technology of China, Thrust of Artificial Intelligence, The Hong Kong University of Science and Technology (Guangzhou), Rutgers University, Department of Computer Science and Engineering. The Hong Kong University of Science and Technology liyichencly@@gmii. com, {owen _ae, urara] ustc. edu. en, yingsthkust-gz. edu, cn, yuyang yeerryyerrrrrrrr edu, xi zionghui@ust.. hk
Abstract
Recently, multi-modal entity alignment has emerged pivotal endeavor for the integration Multi-Modal Knowledge Graphs (MMKGs) originating from diverse data sources. Existing works primarily focus on fully depicting entity features by designing various modality encoders or fusion approaches. However, uncertain correspondences between inter-modal or intra-modal cues, such as weak inter-modal associations, description diversity, and modality absence, still severely hinder the effective exploration of aligned entity similarities. To this end, in this paper, we propose a novel Tackling uncertain correspondences method for Multi-modal Entity Alignment (TMEA). Specifically, handle diverse attribute knowledge descriptions, we design alignment-augmented abstract representation that incorporates the large language model and in-context learning attribute alignment and filtering for generating and embedding the attribute abstract. In order to mitigate the influence of the modality absence, we propose unify all modality features into shared latent subspace and generate pseudo features via variational autoencoders according to existing modal features. Then, we develop an inter-modal commonality enhancement mechanism based on cross-attention with orthogonal constraints, address weak semantic associations between modalities. Extensive experiments two real-world datasets validate the effectiveness of TMEA with a clear improvement over competitive baselines.
Introduction
Multi-Modal Knowledge Graphs (MMKGs) effectively store a substantial volume of knowledge encompassing diverse modalities including visual, relational, and attribute information in structured and organized manner [23
43
44]. They greatly facilitate the advancement of various downstream
tasks such as recommender systems [4]
12 34], video understanding 40 51 39,, and domain- In order to construct a more comprehensive multi-modal knowledge base for supporting external knowledge comprehension and reasoning, the demand for integrating MMKGs from heterogeneous data sources has become extremely urgent. As a result, multi-modal entity alignment match entities referring to identical real-world concepts from distinct MMKGs has emerged as an increasingly critical task for their integration 23 6).
47
specific applications 7
30
35
49].
In the literature, early investigations into entity alignment primarily rely on translational models [9] 53 31 or graph neural networks [ capture semantic relatedness of aligned entities through the geometrical structures in embedding space. However, these methods were originally devised for
"Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

traditional knowledge graphs and struggle to accommodate diverse multi-modal knowl- edge, which makes difficult to fulfill the need for align- ing entities across MMKGs. Therefore, recent studies [5] 21 [9] have introduced di- verse modality encoders fusion approaches tailored to the characteristics of multi- modal knowledge, striving to fully extract information from all modalities so as to effec- tively portray the features of entities. For modality en- coders, MSNEA [6] devel- ops vision-guided encoders for relational and attribute knowledge to achieve inter- modal enhancement. ACK- MMEA [19] unifies the en- coding of multi mudti-modal fea- tures by designing specialized graph neural network to ensure consistent modal aggregation. Regarding fusion approaches, MMEA [5] synthesizes multiple representations by transferring each uni-modal embedding into common space. EVA [22] incorporates multi-modal features into combined embedding using an attention-driven modality weighting mechanism.
MMKG;
MMKG2
117
2008-11-17
2008.916
Release
Runtine
Duration
5.4
Review
Debut
117
Date
Score
Date
Screen Time
Total
Twilight (Film)
Twillght
Leading
Leading
Character
Autor
Actres
Actress
Robert Pattinson
Collaborator
Kristen
Performer
Bella
Stewart
Swan
KSteew
Weight
Height
165
Aligned
Relational Knowledge
A Attribute
Visual
Entity Pair
Kedgl
Know ledge
Figure I: A toy example to illustrate the uncertain correspondences in multi-modal entity alignment, where entities are highlighted with a yellow background.
Although these methods have made remarkable progress in extracting and fusing multi-modal features for entity representation, there still remain many substantial challenges, particularly the limitation of uncertain correspondences between modal or intra-modal cues of entities [33] [6]. Figure illustrates toy example of such uncertain correspondences. Firstly, the weak semantic associations among diverse modal knowledge of entities pose considerable challenge in inter-modal enhancement. In MMMGG, "Twilight" has attributes "Release Date" and "Runtime Duration" along with an image of movie poster containing two roles. The poster does not include content that could semantically overlap with these attributes, which makes it hard to capture their feature commonality to perform mnter-modal enhancement. Secondly, the distinct descriptive manners of entities across different MMKGs complicate the matching of attributes with the same meanings between aligned entities. For instance, "Release Date" for "Twilight" in MMGG, and "Debut Date" for "Twilight (film)" in MMKG2 both mean the initial release date of the movic. This requires a more profound semantic understanding to ascertain if they convey the same meanings. Thirdly, MMKGs often lack complete entity knowledge, with certain modalities entirely devoid of data. This impedes the differentiation of whether two entities aligned, this distinction can be readily misled by consistent modality features of unaligned entities. Due to missing attributes for "Kristen Stewart" M KG1 and images for "KStew"" in MMKG2, there is high chance of erroneous matching "Bella Swan" in KMG based on its visual similarity to "Kristen Stewart". Given the challenges presented above, the exploration of tackling uncertain correspondences crucial in multi-modal entity alignment task.
To address these challenges, in this paper, we propose a novel Tackling uncertain correspondences method for Multi-modal Entity Alignment (TMEA). Specifically, we encode relational, attribute, and visual knowledge into their preliminary feature representations (MKE) module. In MKE, to handle diverse attribute knowledge descriptions, iiigmment-augmenttee abstract representation that incorporates the Large Language Model (25 (LLM) and in-context learning [33 into attribute alignment and filtering for generating and embedding the attribute abstract. To mitigate the impact of the modality absence, we devise a Missing Modality Imputation (MMI) module complete the missing modality by unifying diverse modalities into shared latent subspace and generating pseudo feature via Variational AutoEncoders (VAEs) [18] according existing modal features. For the purpose of addressing the weak semantic associations between modalities, we develop an inter-modal commonality enhancement mechanism based on cross- attention with orthogonal constraints in another Multi-modal Commonality Enhancement (MCE)
Multi-modal Knowledge Encoder
particularly design

---

module. Extensive experiments on two real-world datasets verify the effectiveness and superiority of TMEA with an obvious improvement over competitive baselines] The main contributions of this paper are summarized as follows:
In this paper, we focus on tackling uncertain correspondences between inter-modal or intra-modal cues of entities for multi-modal entity alignment, including weak inter-modal associations, description diversity, and modality missing.
We propose a novel TMEA model to address these three issues, by specially designing the inter-modal commonality enhancement mechanism, alignment-augmenter abstract representation, and pseudo feature generation.
We conduct extensive experiments on two real-world datasets, FB FBI5K-DBI5K and FBISK- YG15K. The experimental results clearly demonstrate the effectiveness and superiority of TMEA, with at least a 32.8% increase in Hits@1 over competitive baselines.
2
Related Work
In this section, we introduce the related research categorized into traditional entity alignment and multi-modal entity alignment.
Traditional Entity Alignment. Traditional knowledge graphs are composed of textual symbol triples, which often result in information overload [4. Therefore, entity alignment was proposed match the identical entities in various KGs for their integration []. Existing methods mainly rely on entity embedding techniques, which can be grouped into triple-based embedding [53] 31] and neighbor-
based embedding [36 42). Triple-based embedding defines an energy function to measure the plausibility of triples. For example, MTransE [9] adopted TransE 2 as the entity encoder to embed different KGs into independent vector spaces and then learned the transitions between aligned entities. IPTransE 533 implemented an iterative strategy and employed it to supplement probably aligned entities into the training set. Neighbor-based embedding exploits the subgraph structure organized by partial relations between entities. For instance, GCN-Align (36] used graph convolutional networks to aggregate the neighborhood information of entities. MiXTEA 42) instructed the graph model learning by an end-to-end mixture teaching of manually labeled mappings and probabilistic pseudo mappings. PEEA [32] increased the connections between far-away entities and labeled ones by incorporating positional information into the representation learning with a position attention layer. However, they cannot accommodate diverse multi-modal knowledge and satisfy the requirement for multi-modal entity alignment.
Multi-Modal Entity Alignment. Due to the urgent demand for the integration of MMKGs, extensive research has been dedicated to multi-modal entity alignment [23 6. 19. Initially, multi-modal entity alignment was regarded special type of link prediction task, aiming to connect entities through the "SameAs" relation 23 Subsequently, the first attempt in multi-mode entity alignment
introduced the pioneering MMEA ( approach, which transferred multi-modal representations into a unified space by minimizing the distance between the fused embeddings and each uni-modal embedding. Later, Liu et al. (22) believed that visual modality has abundant features the advantage, thus leveraging visual similarities to augment the training data by iteratively adding aligned pairs. The previous studies employed independent encoders for feature extraction of each modality, without considering interactions between different modalities. Therefore, MSNEA [6] proposed modality interaction mechanism to guide relationship learning and attribute selection using visual features, and designed multi-modal contrastive learning to make features of aligned entities more similar. Owing to the impact of contextual gaps effectiveness, ACK-MMEA [99 utilized merge and generate operators to build an attribute-consisten MMKG, and aggregated the consistent information through graph neural networks. Despite their remarkable advancement, there still remain some significant challenges like uncertain correspondences between inter-modal or intra-modal cues of entities.
3 Preliminary
Multi-Modal Knowledge Graph (MMKG). A MMKG consists of relational, attribute, and visual knowledge and is formally defined as G (E, A,V.A, P). Here, E, R, I, A, V, T, P denote the sets of entities, relations, images, attributes, attribute values, triples, and entity-image
'The code is available at alatttps//gsthub..cmm//r///

---

Missing Modally Impulation (MME)
VAE Loss
MMKG;
MMKG,
MSE
Modal lodge (MKE)
Relational Koowledge Encoder
VAE Losa
VAE
Auguestation
Transf
Attribate
MSE Las
Trenphate
1100
YAE
Visaa Kanwiegge Enooder
Mulli-Modal Commonality Enbanx ement (MCE)
Transiermer
CrossAAf
CrowAAlt
MMI
MNI
Sub
Commonalls Fahanoment
Commusally Folunremmt
Cammenalls Fnhanouseet
MCE
MCF
MCL Loss
Commonality Enhancement
Figure 2: The framework overview of TMEA.
pairs, respectively. Specifically, the triple set T includes relational and attribute triples, denoted as T TR U TA. where TR {ee, r, et) eh, E &,r E RJ refers to the set of relational triples, and Ta
Ta= = {(e, a, v) e & E A, u E V} represents the set of attribute triples. = {(c, e e E E I} indicates that the image describes the entity
Multi-Modal Entity Alignment (MMEA). We consider the MMEA task between two MMKGs 9i and G2. The objective of MMEA is to pair 1-to-1 corresponding entities that describe identical concepts the real world but exist in separate MMKGs. Represented symbolically, the inputs are two MMKGs 9 (E I,, V1, Tí. P1) and G2 (E2, R2, A2, V2, T2, P2), and the output the set aligned entity pairs S = (()),)) e1 E ε, E E2, E1 = E2], where = means the equivalence of two entities.
Methodology
In this section, we introduce the technical details of our proposed TMEA. As illustrated in Figure 2 TMEA comprises three main modules: 1) Multi-modal Knowledge Encoder (MKE) module to encode relational, attribute, and visual knowledge into their preliminary feature representations, with alignment-augmented abstract representation to address diverse attribute knowledge descriptions: 2) Missing Modality Imputation (MMI) module to unify diverse modalities into shared latent subspace and generate a pseudo feature via VAEs according to existing modal features for completing the missing modality; 3) Multi-modal Commonality Enhancement (MCE) module to enhance semantic associations between modalities by designing an inter- modal commonality enhancement mechanism based cross-attention with orthogonal constraints. For model optimization strategy, integrate multi-modal contrastive learning and bi-directional iteration.
4.1 Multi-Modal Knowledge Encoder
In this module, we encode the relational, attribute, and visual modalities of knowledge acquire their preliminary fcature representations. In particular, to address the diversity of descriptive manners of attribute knowledge, large language model and in-context learning into attribute alignment and filtering for generating and embedding the attribute abstract.
design alignment-augmente abstract representation that incorporates the
Relational Knowledge Encoder. Relational knowledge is composed of a set of relational triples, in the form of (cn, r, ex) E TR. Here, we take TransE (2) with the swapping strategy [5] as the relational knowledge encoder to interpret as a translation vector from en to ét, making relationships captured by simple geometric operations. The score function [en, rx)x) and the margin-based loss LTran are defined as follows:
f(em,T,ez) -1e +r e,
(I)

---

ErranE
Σ
Σ max (0.7+5())-
(2)
TETR
ETR
where ex ef are initial relational feature vectors of еn and et, is the feature embedding of r, ll-Il2 is the Lg norm, ? is the margin hyperparameter, and T is the set of negative examples. By this means, we can obtain the initial relational feature of the entity, denoted as e
Attribute Knowledge Encoder. An attribute triple can be denoted as (e, a, E TA Aligning attributes across different MMKGs is challenging due to the variations in the expression of identical attributes. It necessitates more than a basic lexical comparison and requires a profound semantic understanding to ascertain whether attributes with different labels indeed represent the same concept. Recently, the powerful semantic understanding capabilities of LLMs have been proven across multiple tasks [ 8 [0]. The pre-existing knowledge within LLMs can help bridge gaps in information that is a prompt engineering method included in the prompt. With in-context learning, off the--shhhfff LLM can solve attribute alignment without fine-tuning. Therefore, we design an -aggmeeet- abstract representation that incorporates the GPT 3.5 [25] and in-context learning [13] into attribute alignment and filtering for generating and embedding the attribute abstract. For attribute alignment, the prompt is devised shown in Appendix [A] As the attributes that cannot be aligned often become noise [45], we only retain those attributes that can be aligned according to the results of GPT 3.5. Next, we convert the filtered attribute triples into textual abstract sentence. For entity and its filtered attribute triples (e, a1 v1), (e, d2, U2), ..., (e, an Un the template of abstract sentence S is "a, is vl, 02 is v2, .... On is U,," ". Given two attribute triples ("Twilight", "Release Date", "2008.916") and ("Twilight", "Runtime Duration", "117"" related to "Twilight", we construct a abstract sentence of "Release Date is 2008.9 Runtime Duration is 117". Then, we use a pre-trained BERT (I]] model to generate the representation of this sentence, thus forming the attribute embedding of the entity in the following:
may not be explicitly stated. Moreover, in-context learning 37 52
where task demonstrations
e
WA- n2 j=1
(wR)(W;) + ba,
where wy denotes the j-th word in S, (ERT(-) represents the hidden feature vector in the last layer of BERT, WA A is the weight matrix, and b is the bias vector.
Visual Knowledge Encoder. Visual Enowledge is typically comprised of entity-image pairs, denoted as (e. i) E P. We employ a pre-trained Vision Transformer (ViT) 14 to extract image features. Specifically, the last fully connected layer and softmax layer are removed to acquire the image embeddings of entities. The initial visual feature embedding e for (e, i) is generated as follows:
e² Wi)iT(() + br,
where ViT(.) is the backbone of ViT, and Wz, by are the weight matrix and bias vector.
4.2 Missing Modality Imputation
In real-world scenarios, entities in MMKGs often lack the knowledge of certain modalities. This omission can result from limitations in data collection methods or source availability. Incomplete modal knowledge will potentially lead to erroneous alignment. Effectively addressing and compensat- ing for these missing modalities to guarantee accurate integration are able to improve the robustness of entity alignment process.
In this module, we intend to complete the missing modality by unifying the diverse modalities into a shared latent subspace and generating pseudo feature via Variational AutoEncoders (VAEs) [18] 28
according to existing modal features. VAES can learn latent representations of data, capturing essential structure and distribution. We utilize VAEs to unify features of one target modality and concatenated features of other modalities into a shared latent subspace. In this latent subspace, data from various modalities become comparable. We use the latent representation of concatenated features derived from other modalities to approximate that of the target modality, thereby generating pseudo-features.
Typically, entities in MMKGs cannot lack relational knowledge, because entities without relations are isolated nodes, rendering them devoid of the graph's 's interconnected significance. Therefore, we aim to complete the visual and attribute knowledge for entities. Specifically, we first reduce the dimension of concatenated features of other modalities as follows:
g
RI W Concat(e" R ee bga
(5)
5

---

g²a= = Wg
gi g::Ccaatee ,ea) + bos.
(6)
where Concat(-) is the concatenation operation, Wgt. by. are projection matrix and bias vector. Then, multiple VAEs to learn the latent representations. For each VAE, X is feature of target modality, E Z is the representation of x in the latent space, qa (z xx) is a probabilistic
encoder, and po [ 2) is probabilistic decoder. The loss of VABs designed as follows:
AE X) =
[log Po (x|z)) - DEL (go (2|x) IIpo (z)),
= LvAB Lv Av AV + (VAB(GRA))
(8)
where (+ is the KI. divergence, E is the set of entity features of modality *, and GR is the set of concatenated features of relation and modality *, To unify features of one target modality and concatenated features of other modalities into shared latent subspace, we minimize the distance of their latent representations as follows:
mmee = MSE(ZA, RRR) + MSE(ZZ (ASE(ZZZZRA)) 2
where M.E(() is the mean-square error. Through this operation, we can use Z RI to approximate ZA, and Z to approximate Z In the generation stage, we exploit the results of VAEs to complete the missing modality. We input the latent representation z ZRT, z R.A into the decoders pe (e {z)),
pe (e zz
to generate pseudo feature e e² respectively.
4.3 Multi-Modal Commonality Enhancement
Generally, tenuous semantic associations between disparate modal representations of knowledge significantly hamper efforts toward inter-modal enhancement. In multi-modal tasks, semantic ciations between different modalities are often utilized to mutually enhance features across modal- ities [50) (2). If the semantic association is weak, mechanisms designed for interactions between different modalities may negatively impact overall performance, potentially resulting in performance worse than that of a single modality.
In this module, we enhance semantic associations between modalities by specially designing an inter- modal commonality enhancement mechanism based cross- coss-attention with orthogonal constraints, thereby facilitating a more holistic understanding of the entity. Cross-attention architectures have been empirically substantiated to efficiently capture semantic associations between multiple modalities [38 29). Specifically, we consider two distinct modal features, denoted as x and y. x is designated as the query, while y assumes the roles of both key and value within the attention. The attention mechanism computes a weighted sum over y, prioritizing elements that are most relevant to x. This process refines y based on the commonality with x. The attention defined as follows:
QKT
V, headd Att (xW2 w yWy),
AN(Q, K, V)= = =softmar
(10)
Vd
MH(x,y)= Concat (head1,... head,,) Wh + bj,
(11)
where K, V denote the query, key and value respectively, softmar(-) is the activation function, d is the dimension of key, 7) is the number of heads, w w/K wy are projection matrices for j-th attention head, and W,. by are the projection matrix and bias vector.
Building upon this cross- aots-atttnnion, we concurrently extract the commonalities among relational, attribute, and visual modalities to enhance the overall representation. The operation is as follows:
e =uRRER R + WRR eR) + WRAMMIRAAe ,e.)
(12)
e - WAIMHAz AA) WWRRMHA R e44)
(13)
xI+ = WIR (c .F) WIAMHIA(O o²²)
(14)
where w.a. #,# E (R. AI are learnable weights normalized by the Softmax, and M MH. is the result of multi-head cross anteettttnnionn for commonality extraction between two modalities.
cc)
To amplify the commonalities among features, we additionally design an orthogonal constraint loss, which ensures the difference calculated by subtracting the commonality features from the original features, dissimilar to the query features. For example, M HRI(e² eR means the refined version of e highly relevant to e". The difference can be calculated as ee -MHRRZ(e e

---

representing the content within e that does not include any elements related to e². Hence, it exhibits an orthogonal relationship with e². The orthogonal constraint loss is defined as follows:
(x,)) = (xy")) Fale". cτ) = -MH,#(c* )²),,, (R,A,I), Lo oFole ,e),, e²) + Forth (Fale", ,e)), + Forth Fort (Fale Fort((F e²), + Forth (Fale ) Forth (ora )
(15)
(16)
where (-)th(-) i is the orthogonal constraint function, Fa(-) is the calculation function of difference, Lo is designed orthogonal constraint loss for commonality enhancement.
In this way, we can accentuate the commonalities between modalities to distill inter-modal correspon- dences for more synergistic and semantically coherent representation.
4.4 Model Optimization
Through the multi modal knowledge learning in the prior three modules, obtain enhanced feature embeddings for relational, visual, and attribute modalities. By directly concatenating the enhanced features, we generate an integrated multi-modal representation of entity follows:
e Concat(e R+ R+, ee, ee,),
(17)
where e is the holistic enhanced entity embedding of entity
To ensure that features of aligned entities from different MMKGs are more closely in the space and make the overall modeling more robust, we employ the multi-modal contrastive learning [6 48) as our main optimization objective. Multi-modal contrastive learning contrasts the feature similarity of positive entity pairs with negative entity pairs, from multiple perspectives of holistic modai representations and each uni-modal feature representation. Positive entity pairs refer to two aligned entities, and negative entity pairs refer to two unaligned entities. The main objective of optimization defined as follows:
LJ(E; EE)
1
(1- (1
Σ
(1-y)d(eii,e)) ez) + maxx(1a d(e e2),
2
(18)
(c1.02)EH
Cmel = Ca EEM) + La E + La LE.
(19)
where Lmol is the main objective of optimization, E {M, R, A,I) denotes the modality, Ej, E the sets of entity feature embeddings of modality for G1 and G2. H is the set of sampled entity pairs including positive and negative samples, d(::)) is the function of cosine similarity, y is the indicator of whether an entity pair be aligned, and Yat is the margin hyperparameter.
Combining all the designed losses above, the overall objective of optimization C is as follows:
L LTransE, Xi + Lmme + Lo X2hmsse
(20)
In training stage, we minimize L and update the model parameters via the backpropagation. Owing to the high cost of manual annotations and the expansive scale of MMKGs, the availability of aligned
entity pairs is frequently insufficient. Therefore, we adopt bi-directional iterative strategy [24] based on the asymmetric nature of alignment directions to boost the training. The specific algorithm of this strategy is described in Appendix B
5 Experiment
In this section, we provide extensive experiments on two real-world datasets. To begin, we offer a brief description of experimental setups. Next, the experimental results and analysis are presented to verify the effectiveness of TMEA. More details and results of experiments are provided in Appendix C
5.1 Experimental Setups
we selected two commonly used public datasets, namely FB FB15KDDB 15K and FB15K-YGISK 2 and allocated 20% of the aligned entity pairs as training data. These datasets contain two pairs of aligned MMKGs with relational, attribute, and visual knowledge.
Datasets. Following previous studies 5
7

---

Table 1: The performance of multi-modaa entity alignment two datasets. The best results indicated in bold, and second- best results are marked with underlines.
FB15K- DB1SK
FB15K-YGISK
Method
H@1
l@55
1@@ 10
MR
MRR
H@@
I@55
H@@1
MR
MRR
IPTransE
0.040
0.112
0.173
387.512
0.086
0.031
0.095
0.144
522.235
0.070
GCN-Align
0.043
0.110
0.155
810.648
0.082
0.023
0.072
0.107
1109.845
0.053
BootEA
0.323
0.499
0.579
205.532
0.410
0.234
0.374
0.445
272.120
0.307
SEA
0.170
0.335
0,425
191.903
0.255
0.141
0.287
0.371
207.236
0.218
RAC
0.203
0.360
0.432
453.313
0.281
0.151
0.281
0.345
501.795
0.216
PEEA
0.143
0.254
0.299
1265.616
0.198
0.126
0.223
0.268
989.414
0.175
MixTEA
0.115
0.233
0.299
340.733
0.178
0,085
0.177
0.234
350.439
0.136
PoH.
0.120
0.256
0.167
0.109
0.241
0.154
MMEA
0.265
0.451
0.541
124.807
0.357
0.234
0.398
0.480
147.441
0.317
EVA
0.556
0.666
0.716
139.995
0.609
0.103
0.217
0.278
616.789
0.164
MSNEA
0.653
0.768
0.812
54.025
0.708
0.443
0.626
0.698
85.074
0.529
MCLEA
0,441
0.640
0,710
84,628
0.534
0,406
0.579
0.645
123,394
0,488
ACK-MMEA
0,304
0,549
0.387
0.289
0,496
0,360
MoAlign
0.318
0.564
0.409
0.296
0.525
0.378
TMEA
0.867
0.929
0.944
26.343
0.895
0.818
0.891
0.916
32.864
0.853
Evaluation Metrics. We assessed the alignment probability between entities from different MMKGs based on cosine similarity calculation. Then, we chose H@N (Hits@ N). MRR (Mean Reciprocal Rank), and MR (Mean Rank) metrics evaluate all the models. A higher value for H@ N and MRR indicates better performance, whereas a lower value for MR suggests the same.
Compared Methods We benchmarked TMEA against a selection of both representative and state-of- the-art approaches. For traditional KGs: IPTransE [53], GCN-Align [36], BootEA 31, SEA 26),
RAC [46), PEEA (32), and MiXTEA 42 For MMKGx: PoE 23 .MEE 5 EVA 22
MSNEA (6). MCLEA 21) ACK-MMAA
19 and MoAlign 20
5.2
Performance Comparison
We compared TMEA with several representative and state-of-the-art baselines for multi-modal entity alignment task on FB 1B-55D 1BB55 and FB15K-YG15K datasets. Table the performance of all methods with 20% aligned entity pairs for training. Overall, our proposed TMEA achieves the best performance both datasets. Specifically, we have the following findings. First, TMEA exhibits significant improvements with notable margin compared to not only traditional but also multi- modal methods. Compared the strongest traditional alignment method, BootEA, TMEA achieves at least a 168.4%, 63.0%, and 1 11833 improvement in H@ 1, H@ 10, and MRR on two datasets, respectively. When assessed against the leading modal method, MSNEA, TMEA attained at least a 32.8%. 16.3%, and 26.4% increase in H@1 1, H@ 10, and MRR, respectively. Besides, test the statistical significance and the results suggest TMEA has significant improvements (p-value < 0.001) over baselines. Second, comparing multi-modal entity alignment methods to traditional ones, we observe clear advantage in favor of multi-modal approaches in most cases. TMEA, MSNEA, and MCLEA surpass all traditional approaches, which suggests the effect of leveraging the rich and varied information that multi-modal knowledge provides. This additional context and modeling allow for a more comprehensive depiction of entity features, facilitating more accurate determination of potential alignments. Third, our TMEA method, leveraging only 20% of aligned entity pairs for training, achieves remarkable H @ 10 score of 0.944 and 0.916 on two datasets. This large margin of superiority over other multi-modal entity alignment methods can be attributed to the considering and effective tackling of uncertain correspondences in the alignment process. By addressing these inherent issues, TMEA is demonstrated enhance the precision and reliability of entity alignment for MMKGs.
5.3 Ablation Study
To verify the influence each part in TMEA, we conducted two sets of ablation studies, one set for modality ablation and another for component ablation. The variants for modality ablation are named w/o V. w/o R, and w/o A, corresponding to the removal of visual, relational, and attribute knowledge respectively. In terms of component ablation, w/o MMI and w/o MCE refer to the elimination of
8

---

OMSNNA ETMRA
1.0
1.0
0.9
0.8
0.8
MRR
Hi01
MRR
0.8
0.6
0.4
0.4
40%
Visual Modality
Visual Modality
Attribute Modality
Attribute Modulity
The performance of TMEA and the strongest baseline, MSNEA, when varying ratios of entities have visual or attribute modalities on FB15K-DBI5K.
Figure 3:
modules MMI and MCE, w/o L. and w/o Lmse are the removal of constraint losses, w/o AP denotes the absence of preprocessing for attribute knowledge, and w/o IT indicates the exclusion of the iterative strategy. Table 2 shows the re- sults of ablation study, demonstrating the effectiveness of each part in TMEA. In the results of the modality ablation study, it is evident across both datasets that each modality's 's features apparently contribute to the improvement of entity alignment performance. Furthermore, the most ef- fective modality varies across datasets, which may depend on factors such data sparsity or noise. Regarding the re- sults of component ablation, the effectiveness of each designed component is validated. First, the considerable drop in performance without MCE module (w/o MCE) underscores the capability of MCE module to bolster the commonality among modalities, mitigating the impact of weak semantic associations. Second, the marked deterioration in performance without attribute preprocessing (w/o AP) verifies the effect of integrating the LLM and in-context learning into filtering and abstract sentence embedding. Third, removing MMI module (w/o MMI) leads to an increase in the MR metric, and the effect is different on two datasets, indicating that the effectiveness of MMI module is contingent on the proportion of missing modalities within the dataset. Last, the absence of the iterative strategy (w/o IT) highlights the strategy's role in boosting the robustness of the model, as evidenced by a marked deterioration in MR upon its removal.
Table 2: The ablation study to verify the impact of modal- ity and component on two datasets.
FB15K-DB15K
FBISK- Y
Method
H@1
H@10
MR
H@1
H@ 10
MR
TMFA
0.867
0.944
26.343
0.818
0.916
32.864
w/o V
0.446
0.684
227.984
0.609
0.772
139.901
w/o R
0.630
0.742
290.519
0.657
0.834
96.719
w/o A
0.793
0.904
52,959
0.569
0.735
157.695
w/o AP
0.786
0,903
45.263
0.593
0.757
138.562
w/o MMI
0.852
0,935
32.867
0.779
0.885
56.569
w/o Lm
0.841
0,922
37.829
0.797
0.900
47,584
w/o MCE
0.749
0.868
48.314
0.636
0.809
51.573
w/o Lo
0.842
0.931
30.239
0.794
0.904
44,741
w/o IT
0.728
0.834
64.875
0.645
0.825
112,248
5.4 Modality Sensitivity Analysis
In MMKGs, entities often lack certain modalities of knowledge. Therefore, we address the missing modalities for entities in MMI module. To validate the ability of TMEA to maintain robust perfor- mance with missing modalities, varying ratios of missing modal knowledge. Figure presents the results of TMEA and MSNEA under scenarios with 20%, 40%, 60%, 80%. and 100% entities having visual and attribute modalities on FB15K-DB 15K, respectively. We observe that TMEA outperforms MSNEA at different ratios obviously. In scenarios with missing visual modality. TMEA exhibits slower performance decline compared to MSNEA. Moreover, when only 20% of entities have visual modality, TMEA achieves an MRR exceeding 0.5, while MSNEA falls below 0.4. For missing attribute modality, both TMEA and MSNEA show a relatively minor performance decline. With only 20% of entities having attribute modality, TMEA achieves MRR and H@1 close to 0.8. MSNEA remains relatively unchanged across different proportions due to its insufficient attribute learning of adaptive weights, even with a substantial number of attribute features that only yield marginal improvements. This also indicates that our attribute knowledge encoder more efficiently and fully leverages the semantics of attributes.
compared TMEA with the strongest baseline, MSNEA, under
5.5 Label Dependency Analysis
The availability of aligned entity pairs is frequently insufficient. Importantly, the effective and sufficient exploitation of modti-modal knowledge can significantly mitigate the dependency of methods on alignment labels during training. For the purpose of assessing these methods' reliance on the

---

MFOF
I MMEA
BEVA
EN MINNEA
MCLEA
DACK-MMRA
T. MMoAlggg OTMEA
01.8
0.5
0.8
1681
0.6
MRH
0.6
0.2
0..0
0.0
0.9
20%
50%
80%
20%
50%
90%
FRISK-DBISK
1.0
01.8
00.8
0.81
0.6
0.6
MNR
0.6
0.2
0.0
0.0
0.9
50%
FB15K-YGISK
Figure 4: Comparison results with different ratios alignment labels.
quantity of aligned entity pairs, we conducted series of experiments to train the models using 20% 50%, and 80% aligned entity pairs. Figure shows the comparison results of all multi-modal entity alignment methods. From the overview, TMEA provides an obviously leading performance with different proportions training data. Only TMEA achieves H@ I, H@@ 10, and MRR scores above 0.8 across various percentages. Moreover, its performance does not show a clear decline even with reduction training data proportions, unlike the majority of methods that experience a marked loss in performance. For instance, ACK-MMEA and MoAlign achieve lower H@@ 10 and MRR scores than MCLEA with 50% training data, yet surpass MCLEA when the training data is increased to YG15K. This indicates their strong dependency aligned sample pairs reach 80%, even though the performance of TMEA and MSNEA on the H@ 10 metric appears to be close, there is still noticeable difference in MRR and H@) This suggests that MSNEA performs poorly in aligning difficult samples.
80% on
aligned entity pairs. When the
Conclusion
In this paper, we proposed a novel TMEA to tackle uncertain correspondences between inter-modal or intra-modal cues of entities for modal Knowledge Encoder (MKE) module to encode relational, attribute, and visual knowledge into their preliminary feature representations. In MKE, to address the diversity of descriptive manners of attribute knowledge, we particularly designed alignment-augmentee abstract representation that incorporated the LLM and in-context learning into attribute alignment and filtering for generating and embedding the attribute abstract. To mitigate the influence of the modality absence, we devised a Missing Modality Imputation (MMI) module to complete the missing modality by unifying diverse modalities into a shared latent subspace and generating pscudo feature via VAEs based on existing modal features. For enhancing the semantic associations between modalities, invented inter- modal commonality enhancement mechanism based on attentioon with orthogonal constraints in Multi-modal Commonality Enhancement (MCE) module. Finally, extensive experiments on two real- world datasets were conducted to verify the effectiveness of TMEA with significant improvement over competitive baselines.
entity alignment. Specifically, first developed Multi-
Acknowledgments and Disclosure of Funding
This work was supported in part by the National Natural Science Foundation of China (Grant No. 923 70204), the National Key Research and Development Program of China (Grant No. 2023YFF07250 01), the Guangzhou-HKing Bureau of Guangzhou Municipality, the Guangdong Basic and Applied Basic Research Foundation (Grant No. 2024A1515011839)
(GZ) Joint Funding Program (Grant No. 2023A03J0008), the Education
References
[1] OpenAl: Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Tlge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red
10

---

Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, et al. Gpt-4 technical report, 2023.
[2] Antoine Bordes, Nicolas Usunier, Alberto Garcia-Durai Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In Advances in neural information processing systems, pages 2787-2795, 2013.
[3] Yixin Cao, Zhiyuan Liu, Chengjiang Li, Juanzi Li, and Tat-Seng Chua. Multi-channel graph neu- ral network for entity alignment. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pages 1452-1461, 2019.
[4] Liyi Chen, Zhi Li, Weidong He, Gong Cheng, Tong Xu, Nicholas Jing Yuan, and Enhong Chen. Entity summarization via exploiting description complementarity and salience. IEEE Transactions on Neural Networks and Learning Systems, 34(11):8297-8309 2023.
[5] Liyi Chen, Zhi Li, Yijun Wang, Tong Xu, Zhefeng Wang, and Enhong Chen. Mmea: Entity alignment for multi-modal knowledge graph. In International Conference on Knowledge Science, Engineering and Management, pages 134-147. Springer, 2020.
[6] Liyi Chen, Zhi Li, Tong Xu, Han Wu, Zhefeng Wang, Nicholas Jing Yuan, and Enhong Chen. Multi-modal siamese network for entity alignment. In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining, pages 118-126, 2022.
[7] Liyi Chen, Chuan Qin, Ying Sun, Xin Song, Tong Xu, Hengshu Zhu, and Hui Xiong. Collaboration-awart hybrid learning for knowledge development prediction. In Proceedings of the ACM on Web Conference 2024, pages 3976-3985 2024.
[8] Liyi Chen, Panrong Tong, Zhongming Jin, Ying Sun, Jieping Yc, and Hui Xiong. Plan-on Plan-on-graph: Self-correcting adaptive planning of large language model knowledge graphs. In Proceedings of the 38th Conference on Neural Information Processing Systems, 2024.
[9] Muhao Chen, Yingtao Tian, Mohan Yang. and Carlo Zaniolo. Multilingual knowledge graph embeddings for cross-lingual knowledge alignment. Proceedings of the 26th International Joint Conference Artificial Intelligence, page 1511-1517, 2017.
[10] Xi Chen, Xinjiang Lu, Haoran Xin, Wenjun Peng, Haoyang Duan, Feihu Jiang, Jingbo Zhou, and Hui Xiong. A table-to-text framework with heterogencous multidominance attention and self-evaluated multi-pass deliberation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 607-620, 2023.
[11] Jacob Devlin, Ming-Woo Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171 4171-4186. 2019.
[12] Leilei Ding, Dazhong Shen, Chao Wang, Tianfu Wang, Le Zhang, Hui Xiong, and Yanyong Zhang. Dgr: A general graph desmoothing framework for recommendation via global and local perspectives. In Proceedings of the 33th International Joint Conference on Artificial Intelligence, 2024.
[13] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey on in-context learning, arXiv preprini arXiv:2301.0000 2022.
[14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, 2021.
[15] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedfor- ward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256. JMLR Workshop and Conference Proceedings, 2010.
11

---

[16] Yatai Ji, Junjie Wang, Yuan Gong, Lin Zhang, Yanru Zhu, Hongfa Wang, Jiaxing Zhang, Tetsuya Sakai, and Yujuu Yang. Map: Multimodal uncertainty-awe awware Aasounlanguage pre pre-training model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23262-23271, 2023.
[17] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations, 2015.
[18] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Proceedings of the 2nd International Conference on Learning Representations, 2014.
[19] Qian Li, Shu Guo, Yangyifei Luo, Cheng Ji, Lihong Wang. Jiawei Sheng, and Jianxin Li. Attribute-consistent knowledge graph representation learning for multi-modal entity alignment. In Proceedings of the ACM Web Conference 2023, pages 2499-2508, 2023.
[20] Qian Li, Cheng Ji, Shu Guo, Zhaoji Liang, Lihong Wang, and Jianxin Li. Multi-modal knowledge graph transformer framework for multi-modas entity alignment. In Findings of the Association for Compulational Linguistics: EMNLP 2023, pages 987-999, 2023.
[21] Zhenxi Lin, Ziheng Zhang, Meng Wang, Yinghui Shi. Xian Wu, and Yefeng Zheng. Multi- modal contrastive representation learning for entity alignment. In Proceedings of the 29th International Conference on Computational Linguistics, pages 2572-2584, 2022.
[22] Fangyu Liu, Muhao Chen, Dan Roth, and Nigel Collier. Visual pivoting for (unsupervised) entity alignment. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2021.
[23] Ye Liu, Hui Li, Alberto Garcia-Duran, Mathias Niepert, Daniel Onoro-Rubio, and David Rosenblum. Mmkg: multi-modal knowledge graphs. In European Semantic Web Conference, pages 459-474. Springer, 2019.
[24] Xin Mao, Wenting Wang, Huimin Xu, Man Lan, and Yuanbin Wu. Mraea: An efficient and robust entity alignment approach for cross- lonns-l knowledge graph. In Proceedings of the 13th International Conference on Web Search and Data Mining, page 420 428, 2020.
[25] OpenAl. Introducing chatgpt. https://openat: cww coo/index/ccie chattptt// 2022. Accessed: 2022-11-30.
[26] Shichao Pei, Lu Yu, Robert Hoehndorf, and Xiangliang Zhang. Semi-supervised entity align- ment via knowledge graph embedding with awareness of degree difference. In The World Wide Web Conference, pages 31 3130-3136, 2019.
[27] Leigang Qu, Meng Liu, Jianlong Wu, Zan Gao, and Liqiang Nie. Dynamic modality interaction modeling for image-text retrieval. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1 1104-1110 1113, 2021.
[28] Dazhong Shen, Chuan Qin, Chao Wang. Hengshu Zhu, Enhong Chen, and Hui Xiong. Regular- izing variational autoencoder with diversity and uncertainty awareness. In 30th International Joint Conference on Artificial Intelligence, IJCAI 2021, pages 2964-2977, International Joint Conferences on Artificial Intelligence, 2021.
[29] Yongduo Sui, Wenyu Mao, Shuyao Wang, Xiang Wang, Jiancan Wu, Xiangnan He, and Tat- Seng Chua. Enhancing out-of diittii-ittittoionn generalization on graphs via causal attention learning. ACM Transactions on Knowledge Discovery from Data, 18(5):1-24, 2024.
[30] Ying Sun, Hengshu Zhu, Lu Wang, Le Zhang, and Hui Xiong. Large-scale online job search behaviors reveal labor market shifts amid covid-19. Nature Cities, 1(2):150-163, 2024.
[31] Zequn Sun, Wei Hu, Qingheng Zhang, and Yuzhong Qu. Bootstrapping entity alignment with knowledge graph embedding. In Proceedings of the Twenty- International Joint Conference Artificial Intelligence, pages 4396 4402, 2018
[32] Wei Tang, Fenglong Su, Haifeng Sun, Qi Qi, Jingyu Wang, and Hao Yang. Weakly supervised entity alignment with positional inspiration. Proceedings of the 16th ACM International Conference Web Search and Data Mining, page 814-822, 2023.
12

---

[33] Hu Wang. Jianpeng Zhang, Yuanhong Chen, Congbo Ma, Jodie Avery, Louise Hull, and G. Carneiro. Uncertainty-awer aware multi-modal learning via cross-modal random network predic- tion. In European Conference on Computer Vision 2022.
[34] Shuyao Wang. Yongduo Sui, Jiancan Wu, Zhi Zheng, and Hui Xiong. Dynamic sparse learning: A novel paradigm for efficient recommendation. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, pages 740-749, 2024.
[35] Tianfu Wang, Liwei Deng, Chao Wang, Jianxun Lian, Yue Yan, Nicholas Jing Yuan, Qi Zhang, and Hui Xiong, Comet: Nit price prediction with wallet profiling. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3893-5904, 2024.
[36] Zhichun Wang, Qingsong Lv, Xiaohan Lan, and Yu Zhang. Cross-lingual knowledge graph alignment via graph convolutional networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 149-357, 2018.
[37] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 02444824428872,,--
[38] Xi Wei, Tianzhu Zhang. Yan Li, Yongdong Zhang, and Feng Wu. Multi-modality cross attention network for image and sentence matching. In Computer Vision and Pattern Recognition, 2020.
[39] Shiwei Wu, Joya Chen, Kevin Qinghong Lin, Qimeng Wang. Yan Gao, Qianli Xu, Tong Xu, Yao Hu, Enhong Chen, and Mike Zheng Shou. Videollm-mod: Efficient video-language streaming with mixture-of-depths vision computation. In Proceedings of the 38th Conference on Neural Information Processing Systems, 2024.
[40] Shiwei Wu, Joya Chen, Tong Xu, Liyi Chen, Lingfei Wu, Yao Hu, and Enhong Chen. Linking the characters: Video-oriented social graph generation via hierarchical-cumullttee gen. In Proceedings of the 29th ACM International Conference on Multimedia, pages 4716 47244, 2021.
[41] Wei Wu, Chao Wang. Dazhong Shen, Chuan Qin, Liyi Chen, and Hui Xiong. Afdgef: Adaptive feature de-correlation graph collaborative filtering for recommendations. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1242-1252, 2024.
[42] Feng Xie, Xin Song, Xiang Zeng, Xuechen Zhao, Lei Tian, Bin Zhou, and Yusong Tan. Mixtea: Supi-supe entity alignment with mixture teaching. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 886-896,70 2023.
[43] Derong Xu, Tong Xu, Shiwei Wu, Jingbo Zhou, and Enhong Chen. Relation-enhanced negative sampling for multimodal knowledge graph completion. In Proceedings of the 30th ACM International Conference on Multimedia, pages 3857-3866, 2022.
[44] Derong Xu, Jingbo Zhou, Tong Xu, Yuan Xia, Ji Liu, Enhong Chen, and Dejing Dou. Multi- modal biological knowledge graph completion via triple co-attention mechanism. In 2023 IEEE 39th International Conference on Data Engineering (ICDE), pages 3928-3941. IEEE, 2023.
[45] Hsiu-Wei Yang, Yanyan Zou, Peng Shi, Wei Lu, Jimmy Lin, and Xu Sun. Aligning cross-lingual entities with multi-aspect information. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4430 4440, 2019.
[46] Weixin Zeng, Xiang Zhao, Jiuyang Tang, and Changjun Fan. Reinforced active entity alignment. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 2477-2486, 2021.
[47] He Zhang, Ying Sun, Weiyu Guo, Yafei Liu, Haonan Lu, Xiaodong Lin, and Hui Xiong. Inter- active interior design recommendation via coarse-to-fine multimodal reinforcement learning. In Proceedings of the 31st ACM International Conference Multimedia, pages 6472-6480, 2023.
13

---

[48] Shengzhe Zhang, Liyi Chen, Chao Wang, Shuangli Li, and Hui Xiong. Temporal graph contrastive learning for sequential recommendation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages
[49] Weijia Zhang, Jindong Han, Zhao Xu, Hang Ni, Hao Liu, and Hui Xiong. Urban foundation models: A survey. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 6633-6643, 2024.
[50] Yunhua Zhang, Hazel Doughty, and Cees Snoek. Learning unseen modality interaction. Ad- vances in Neural Information Processing Systems, 36, 2024.
[51] Yuting Zhang, Zhao Zhang, Yiqing Wu, Ying Sun, Fuzhen Zhuang, Wenhui Yu, Lantao Hu, Han Li, Kun Gai, Zhulin An, et al. Tag tree-guided multi goaliie alignment for multi-domain short video recommendation. In ACM Multimedia 2024.
[52] Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoe V Le, et al. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference Learning Representations, 2022.
[53] Hao Zhu, Ruobing Xie, Zhiyuan Liu, and Maosong Sun. Iterative entity alignment via joint knowledge embeddings. In Proceedings of the Twenty-Sixth International Joint Conference Artificial Intelligence, pages 2558-4264, 2017.
14

---

Appendix
A Prompt for Attribute Alignment
Aligning attributes nccessitates a profound semantic understanding to ascertain whether attributes with different literals indeed represent the same concept. Therefore, we incorporate the GPT 3.5 and in-context learning into aligning attributes from two MMKGs. The prompt used for attribute alignment is shown in Table 3
Table 3: The prompt used for attribute alignment.
You are an excellent attribute alignment expert, and now there are two sets of attributes in two knowledge graphs (KGs). You need to align the attributes in both KGs one by one based whether they have the same meaning, but there may attributes that cannot be aligned. Here is an example:
The attributes and their values are
KG1: {(date of birth, 1880-10-12), (latitude, 54.20867775), (longitude, 9.70404945),,
KG2: (ate 1563.333), (activeYears, 1652), (birth Year, 1975), (long, 14.548)) The attributes that can be aligned are (date of birth, birthDate) (longitude, long)
Please align many attributes as possible from the KGs following. Directly output results by line, without outputting duplicate or irrelevant information. Even if there is information that cannot aligned, do not output explanations. The attributes and their values are KGI: [(al, v1), (a2, v2). ...). KG2: ((al, v1), (a2, v2). )..).
B Iterative Strategy Algorithm
We adopt bi-directional iterative strategy 24 based on the asymmetric nature of alignment directions to boost the training. During the current iteration, if the entities e1 E ε1 and E & are the mutual nearest neighbors of each other, they will be treated as a newly aligned pair to be added into the training set for the subsequent iteration. Algorithm I outlines the details of this iterative process.
C More Experimental Details and Results
C.1 Dataset Description
We adopted two representative public datasets commonly used in multi-modal entity alignment
task [23 6 19] namely FB15K-DBISK and FB15K-YGI5K. The statistical details of FB15K-
DB15K and FB 1555KK are presented in Table 4 FBISK is derived from Freebase 3 large-scale knowledge base acquired by Google It is widely applied for knowledge graph completion tasks and evaluating link prediction models. DB15K 15K is subset of DBpedia 5l while YG15K is subset of YAGO The entities in FB15K can be aligned with the entities in DB15K and YGI5K, respectively.
C.2 Metric Explanation
We chose H@N (Hits@N)) MRR (Mean Reciprocal Rank), and MR (Mean Rank) as evaluation metrics. According to similarity ranking, H@N counts the number of times the true aligned entity
http:://ppenaa..mmmbbbb//c/aca.
https://uww.goop/ fow.eeec..oofrrrbeeeeeeeeeeeeeee freebase
httpe://developerrs.u google.
https://yago-kno//// htw.:aggooooooldeeerr..ooo
15

---

Algorithm Bi-DDirectionl Iterative Strategy for MMKGs
Require:
Two MMKGs Gi, G2, set of aligned entity pairs S, & E & &g are entity sets that does not exist in S respectively, E,", E are sets of holistic enhanced entity embedding respectively.
Ensure: Model parameters 0
1: Initialize model parameters
2: repeat
3: Train model on 91, G2, until the loss of validation does not decrease
4: for E ε do
5: Get the embedding eM E EM entity
6:
From Ey, find the entity e' whose feature e/M E is most similar to e.M if e is also most similar to the embedding of entity e from & then
7:
8:
S x U e(),e')
9:
3 - E
10:
Eg + E2
11:
end
12:
end for
13:
until No more aligned entities are added to S
Table 4: Statistics of MMKGs in the dataset.
Statistics
FB15K
DB15K
YG15K
# Entities
14,951
12,842
15,404
# Relations
1,345
279
32
# Attributes
116
225
7
# Relational Triples
592,213
89,197
122,886
# Attribute Triples
29,395
48,080
23,532
# Emtee Pairs
13,444
12,837
11,194
# Aligned Entity Pairs
12,846
11,199
appears within the top-N predicted entities and calculates the average across all source entities. MRR denotes the mean reciprocal rank of correctly aligned entities, and MR represents the mean rank of correctly aligned entities. A higher value for H@N and MRR indicates better performance, whereas a lower value for MR suggests the same. The formulas for calculating these metrics are as follows:
H@N
[rank; S N],
(21)
MRR S
MR
[S] Σ ranki,
(22)
rank;
where S refers to the of aligned entity pairs, rank; denotes the rank of the correctly aligned entity for the i-th query entity, and I [.] is the function indicator.
C.3 Model Configurations
We initialized all weight matrices with Xavier Normal initializer 15 For efficiency, we froze the parameters of pre-trained ViT [14] and BERT [I]] in the training stage. The dimensions of entity features and were 100. In VAEs, the encoder and decoder were both composed of two fully connected layers, and the dimension of the latent representations was set to 64. The number of heads 1. In Let, Tel We used the mini-batch method with a batch size of 5000. The learning rate was 0.001 and Adam optimizer [17] was adopted. Our model was implemented using the framework of PyTorch The experiments were conducted on a server with two Intel Xeon Silver 4214R CPUs 2.40GHz, four NVIDIA GeForce RTX 3090 GPUs, and 256 GB RAM memory.
77 is 2. In CTrans E.
2. In the overall objective, x and X2 were 1c-2.
https://ppytorch.org
16

---

C.4 Baseline Description
selected both representative and state-of-the-art baselines for performance comparison. The descriptions of these baselines are introduced in detail as follows:
For traditional KGs:
I
53 the first to propose an iterative strategy for entity alignment iteratively create newly aligned entity pairs for soft alignment.
GCN-Align 36 leverages the graph convolutional networks to aggregate structural and attribute knowledge of entities.
adopts the iterative strategy for data augmentation and designs an alignment editing method to reduce error accumulation during the process of iterations.
BootEA 31
SEA 26 exploits labeled data and abundant unlabeled data together and learns the difference in degree of nodes with adversarial training.
RAC 46 contrasts different views of entity representations by designing an unsupervised contrastive loss and exploits the unlabeled data augment supervision signals.
PEEA [32 increases the connections between far-away entities and labels ones by incorpo- rating positional information into the representation learning with position attention layer for fine-grained information.
MixTEA 42 instructs the model learning by an end-to-end mixture teaching of manually labeled mappings and probabilistic pseudo mappings.
For MMKGs:
PoE [24] employs Product of Experts approach to calculate the probability of alignment following the generation individual uni-modal representations, subsequently determining the "SameAs" links between aligned entities.
MMEA 5 initially produces entity representations with relational, visual, and numerical features and transfers each uni-modal feature embedding into a unified space to facilitate integration.
EVA 22 adopts ResNet, GCN, feedforward network learn images, graph structure, and triples, and then incorporates multi-modal data into combined embedding using an antention-driven modality weighting.
MSNEA 6 implements inter-modal enhancement that leverages visual features to steer the learning of relational and attribute features, with multi-modal contrastive learning to mitigate the overpowering impact weak modalities.
MCLEA 21
takes into account task-specifes modalities and models the inter-modal re- lationships for each entity representation. It then employs contrastive learning jointly capture both intra-modal and inter-modal interactions.
ACK-MMEA 19 integrates consistent alignment knowledge via merge and generate operators, and leverages graph neural network called ConsisIGNN aggregate consistent information.
MoAlign (20) hierarchically incorporates neighbor features, multi-modal attributes, and entity types by a hierarchical modifiable self-attention block in transformer encoder to maintain the unique semantics of different information.
The results of IPTransE, GCN-Align, SEA, PoE, and MMEA are reported in [5). The results of
BootEA, RAC, EVA, and MSNEA reported in (6) The results of ACK-MMEA and MoAlign reported in (19] and 200 The results of PEEA MiXTEA 9 and MCLEA 10
are reproduced by
running the codes in GitHub with default hyperparameter settings.
17

---

C.5 Parameter Sensitivity
We have introduced coefficients A1 and À2 for the two constraint losses in the overall objective function to balance the impact of different losses. Here, we investigate the performance change of TMEA with varying coefficients. We conducted experiments by choosing values for λ and X2 from the list [lc-5, 1e-4, 1e-3, le-2, le-1, 11, and the results FB15K W1SK-DBIS 15K are illustrated in Figure S The results indicate that the optimal values for Å and X2 are le-2. When values are greater than le-2, the performance shows declining trend. This is because excessive emphasis constraint behavior can lead to a deviation in the optimization direction.
0.905
0.880
0.900
0.875
0.870
MRR
0.895
0.865
0.890
0.860
0.885
0.855
0.880
0.850
1e-5
1e-3 le-2
Ic-5
1c-4
Ic-3
lc-2 lc-1
A1
41
0.905
0,880
0.900
0.875
MRR
0.895
0,870
0.865
0.890
0.860
0.885
0.855
1e-5 1e-3 le-2
1
1e-5
1c-4
le-3
le-2 le-1
Az
A2
Figure 5: The parameter sensitivity FB15K-DBI5K.
D
Broader Impact
The impact of TMEA extends beyond its immediate application in integrating knowledge graphs from diverse data sources. By addressing the challenges of tackling uncertain correspondences, TMEA offers framework that be generalized various multimodal learning tasks like multimodal recommendation systems, multimodal document classification,, and multimodal sentiment analysis.
For instance, in the domain of multi-modal recommendation systems, where diverse sources of information need to be fused to provide personalized recommendations, TMEA can adapted to enhance the alignment between different modalities (e.g., preferences, item descriptions, and user-item interactions). By leveraging its alignment-augmentee abstract representation and inter- modal commonality enhancement mechanism, TMEA can effectively capture the subtle semantic associations between modalities, leading to more accurate and personalized recommendations.
Furthermore, TMEA's approach to handling diverse attribute knowledge descriptions and modality absence can also benefit other multi-modal learning tasks, such as image-text matching, cross- modal retrieval, and multimodal sentiment analysis. In these tasks, where aligning and integrating information from multiple modalities is crucial, TMEA's methodology unifying modality features and enhancing commonalities can contribute to improved performance and robustness.
By fostering an environment where multi-modal data is more accurately aligned and integrated, the TMEA model paves the way for novel applications and research opportunities. For instance, in healthcare, could support more robust diagnostic tools and patient care strategies by providing comprehensive view of patient data across modalities. In financial markets, it might offer deeper insights through the integration of textual reports, market data, and socio-economic indicators, leading to more informed investment strategies. Beyond these, the model's application could extend to areas like intelligent urban planning, environmental monitoring, and personalized education, each benefiting from the nuanced understanding and integration of diverse data streams.
18

---

In summary. TMEA's broader impact lies in its potential to advance the field of multi- modal learning by providing a versatile framework that can be applied a wide range of tasks or scenarios requiring multi-modal fusion and alignment.
E
Limitations
Although the TMEA method offers apparent advantages, its practical application is limited by several factors. Firstly, the attribute knowledge encoder relies the large language model trained general corpora, rather than KG-specific data. Consequently, the model's semantic understanding may lack the depth and specificity necessary for KG contexts. In the future, will explore fine-tuning open-source large language models to enhance semantic understanding specifically tailored to KG attributes. Secondly, TMEA depends aligned entity annotation data for feature extraction and model optimization. This dependency presents challenges, particularly in scenarios where annotations scarce or costly obtain. To address this issue, our future investigations will focus fully leveraging multi-modal data in unsupervised settings, aiming to mitigate the reliance on annotated data while enhancing model performance.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please see the parts of abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in the Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

Justification:
Please see Section !
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please Appendix C.3 and Appendix
C.4
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [Yes]
Justification: Please see the code and datasets in our GitHub.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
caiioooooolcyy lc for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /nippe.cc/pullic/uuiii wwwtii/gguie///eeeeceoocc CoodeSubeiiieeee ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please see Appendix
C.3
C.4
and Section 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Please see Section 5 and Table
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
22

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:: http::::///uriss../em
Answer:
[Yes]
Justification:
We conduct the research with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please Appendix D
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
23

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The datasets are from httt::://githubb.oom//mmmm//ppp
in
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee whworee.wthche....c.. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
24

---

Answer: [Yes]
Justification:
The documentation is provided in our GitHub.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "2Erw4qFNnM", "reviewer": "Reviewer_PXNL", "summary": "This paper addresses the task of aligning entities across multi-modal knowledge graphs (MMKGs). The authors propose a novel method called TMEA to tackle the challenges of uncertain correspondences between inter-modal or intra-modal cues of entities. The TMEA method consists of several key components: 1. Multi-modal Knowledge Encoder: This module encodes relational, attribute, and visual knowledge into preliminary feature representations. It includes an alignment-augmented abstract representation that leverages the Large Language Model for attribute alignment and filtering. 2. Missing Modality Imputation: This module addresses the issue of missing modalities by unifying diverse modalities into a shared latent subspace and generating pseudo features via Variational AutoEncoders. 3. Multi-modal Commonality Enhancement: This module enhances the semantic associations between modalities using a cross-attention mechanism with orthogonal constraints. The experiments validate the effectiveness of the method.", "questions": "Please see the weaknesses above.", "limitations": "YES", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.The problem addressed in this paper is highly practical and represents a pivotal issue for multi-knowledge graph entity alignment.\n2.Extensive expriments. \n3.The method is clearly presented.", "weaknesses": "1. the proposed method seems to be a simple combination of the existing modules. Please clarify the challenges and the impact of this paper.\n2. The authors should elaborate on the specific issues that arise when the semantic relationship between different modalities of the same entity is weak.\n3. The alignment-augmented abstract representation is referenced multiple times in the article \"to address diverse attribute knowledge descriptions.\" However, the article does not delve further into this concept or provide a concrete explanation.\n4. The ablation was conducted with only one variable, and the essential components necessary to achieve a significant performance improvement were not identified. It is vital to explore whether each component, such as the loss function or MMI, contributes linearly and significantly enhances the model. The authors should engage in further discussions and experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your valuable feedback on our paper. We appreciate your recognition of the significance of our research problem, extensive experiments, and clear presentation of the methodology. In response to your concerns, we would like to address the following points:\n\n- **[W1: Module combination & Challenges & Impact]**:\n  We would like to clarify that our proposed method is not just a simple combination of existing modules and we have multiple innovative designs in our method.\n  In this paper, we focus on tackling uncertain correspondences between inter-modal or intra-modal cues of entities for MMEA, which is a crucial problem that has not been solved in previous work.\n  The **challenges** include three points, i.e., weak inter-modal associations, description diversity, and modality missing. More details of these challenges are described in the Introduction. Our technical **novelty** is summarized as follows:\n   - To address weak semantic associations, we design an ***inter-modal commonality enhancement mechanism*** based on cross-attention with ***orthogonal constraints***.\n   - To handle diverse attribute knowledge descriptions, we design an ***alignment-augmented abstract representation*** that incorporates the LLM and in-context learning into attribute alignment and filtering for ***generating and embedding the attribute abstract***.\n   - To mitigate the impact of the modality absence, we propose to ***unify diverse modalities into a shared latent subspace and generate pseudo features*** via VAEs according to existing modal features.\n\n\n    Our experimental results show promising performance, which proves that solving the crucial problem of uncertain correspondences can effectively improve the performance of MMEA.\n\n \n  **Impact**: Our method can address the issue of uncertain correspondences in MMEA task, which enhances the feature learning of multimodal knowledge and more effectively aligns entities for better integration of MMKGs. This provides a more comprehensive external knowledge base for downstream applications such as recommendation systems and question-answering systems, thereby improving their performance. \n  \n- **[W2: Specific issues about weak semantic associations]**:\n  Thank you for your valuable suggestion. We would like to clarify that in multimodal tasks, semantic associations between different modalities are often utilized to mutually enhance features across modalities [1] [2]. If the semantic association is weak, modules designed for interactions between different modalities may negatively impact overall performance, potentially resulting in performance worse than that of a single modality. We will add this content in the revised paper.\n\n  [1] Zhang, Yunhua, Hazel Doughty, and Cees Snoek. \"Learning unseen modality interaction.\" NIPS 2023.\n\n  [2] Qu, Leigang, et al. \"Dynamic modality interaction modeling for image-text retrieval.\" SIGIR 2021.\n\n- **[W3: Explanation of diverse attribute knowledge descriptions]**:\n As we have described in lines 65-69, \"The distinct descriptive manners of entities across different MMKGs\ncomplicate the matching of attributes with the same meanings between aligned entities.\nFor instance,  ``Release Date`` for ``Twilight`` in $MMKG_1$ and ``Debut Date`` for ``Twilight (film)`` in $MMKG_2$ both mean the initial release date of the movie\",\n\"Diverse attribute knowledge descriptions\" refers to instances where the same attribute across two KGs is described using different terms, making it challenging for computing word embedding similarity to recognize that these terms denote the same underlying meaning. In addition to the above example, ``Storage Capacity`` and ``Memory Size`` may also describe the same concept. \n\n- **[W4: The ablation for the loss function and MMI]**:\n  The ablation results for the loss functions (w/o $L_o$ and w/o $L_{mse}$) and MMI (w/o MMI) are shown in Table 2. \n   The essential components in TMEA include MMI module (w/o MMI), MCE module (w/o MCE), alignment-augmented abstract representation (w/o AP), orthogonal constraint loss $L_o$ in MCE module (w/o $L_o$), MSE loss $L_{mse}$ in MMI module (w/o $L_{mse}$), and iterative strategy (w/o IT). \n   In Table 2, we have presented ablation studies to demonstrate the effectiveness of all these components. If there are any further suggestions, we would be glad to supplement the experiments.\n\nWe hope these responses effectively address your concerns. We will make revisions to further clarify these aspects in our revised paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Vl7cf6sWMh", "reviewer": "Reviewer_Sc2m", "summary": "This paper addresses the task of multi-modal entity alignment for integrating MMKGs. Existing efforts mostly focus on capturing entity features via diverse modality encoders or fusion methods but face issues with uncertain correspondences. To overcome these challenges, the authors propose a novel method called TMEA. TMEA introduces alignment-augmented abstract representation using a large language model (LLM) and in-context learning to manage diverse attribute descriptions. It also mitigates modality absence by unifying all modality features into a shared latent subspace and generating pseudo features with variational autoencoders (VAEs). Furthermore, it addresses weak inter-modal associations through an inter-modal commonality enhancement mechanism based on cross-attention with orthogonal constraints. Extensive experiments on two real-world datasets demonstrate the effectiveness of TMEA, showing significant improvements over baseline methods.", "questions": "- Could the authors provide a more explicit summary of the contributions of this paper?\n- How have the authors considered addressing attribute learning issues in different knowledge graphs, especially those with specialized knowledge?\n- Could the authors plan to release their code?", "limitations": "The authors have adequately addressed the limitations.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This paper proposes to address the issues of uncertain correspondences for MMEA, such as weak inter-modal associations, description diversity, and modality absence. It is a novel view to improve MMEA task by exploring the similarities of aligned entities.\n- This paper proposes a novel method named TMEA. TMEA includes alignment-augmented abstract representation to handle diverse attribute knowledge descriptions, shared latent subspace for missing modality imputation, and inter-modal commonality enhancement mechanism to enhance modality associations.\n- Extensive experiments on two real-world datasets are conducted to validate the effectiveness of TMEA, showing clear improvements over competitive baselines.", "weaknesses": "- The contributions of the paper are not explicitly summarized. It appears that the primary contribution is the design of new structures to address uncertain correspondences, rather than focusing on multi-modal encoding.\n- General models like ChatGPT can sometimes have difficulty understanding the specific language and context within knowledge graphs. Have the authors considered personalized attribute learning methods that could more effectively grasp the semantics for different knowledge graphs?\n- Could the authors clarify whether they plan to release their code? This would greatly benefit the research community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your valuable feedback on our paper. We appreciate your recognition of \nthe novelty of our research problem and methodology, as well as the robustness of our experiments. In response to your concerns, we would like to address the following points:\n\n- **[W1 & Q1: Summary of contributions]**: Thank you for your insightful suggestion. Below, we provide a more explicit summary of our contributions:\n\n  - In this paper, we focus on tackling uncertain correspondences between inter-modal or intra-modal cues of entities for multi-modal entity alignment, including weak inter-modal associations, description diversity, and modality missing.\n  - We propose a novel TMEA model to address these three issues by specially designing an inter-modal commonality enhancement mechanism based on cross-attention with orthogonal constraints, designing an alignment-augmented abstract representation that incorporates the LLM and in-context learning into attribute alignment, and unifying diverse modalities into a shared latent subspace and generating pseudo features via VAEs according to existing modal features.\n  - We conduct extensive experiments on two real-world datasets, FB15K-DB15K and FB15K-YG15K. The experimental results clearly demonstrate the effectiveness and superiority of our proposed TMEA, showing a significant improvement over competitive baselines, with at least a 32.8\\% increase in Hits@1.\n\n  Furthermore, we will add this content into the revised paper.\n\n- **[W2 & Q2: Specialized knowledge]**:\nRegarding your concern about specialized knowledge, we agree that it is indeed an excellent research problem.\n  Currently, general LLMs can handle a lot of specialized knowledge such as law and medicine, but there are still limitations. In future work, we will focus more on the alignment of specialized KGs. In Appendix E, we propose exploring the fine-tuning of open-source LLMs to enhance semantic understanding. However, the main focus of our work remains on addressing the issue of uncertain correspondences in MMEA task.\n\n- **[W3 & Q3: Code release]**: Thank you for your constructive suggestion. To ensure reproducibility and promote research in this field, we plan to release our code. We stated in the checklist that we would release the code upon publication.\n\n\nWe hope these responses effectively address your concerns. We will make revisions to further clarify these aspects in our revised paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rI2mGF4yBq", "reviewer": "Reviewer_qUrF", "summary": "This paper proposed a novel method for tackling uncertain correspondences in multi-modal entity alignment, called TMEA. The approach addressed challenges such as weak inter-modal associations, description diversity, and modality absence that hinder effective entity similarity exploration. TMEA employd alignment-augmented abstract representation, integrating large language models and in-context learning to enhance attribute alignment and filtering. To address modality absence, it unified all modality features into a shared latent subspace and generates pseudo features using variational autoencoders. Additionally, it introduced an inter-modal commonality enhancement mechanism based on cross-attention with orthogonal constraints to improve weak semantic associations. Extensive experiments on two real-world datasets has demonstrated the effectiveness of TMEA, showing significant improvements over competitive baselines.", "questions": "Why should we consider modal missingness, the Missing Modality Imputation(MMI) module doesn't seem to play a big role?", "limitations": "The authors have adequately addressed the limitations and potential negative societal impacts of their work in Appendix E. They underscored limitations in semantic specificity stemming from reliance on language models trained on general corpora and dependence on annotated data, which hampers practical application in knowledge graph contexts. Future research aims to refine these aspects for broader applicability in unsupervised settings within KGs.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper effectively encoded relational, attribute, and visual knowledge into feature representations using a Multi-modal Knowledge Encoder   (MKE) module, providing a holistic view of entity features.\n2. This paper used a Large Language Model (LLM) and in-context learning for better attribute alignment and filtering, and addressed modality absence through the Missing Modality Imputation (MMI) module, which generated pseudo features using Variational AutoEncoders (VAEs).\n3. This paper developed an inter-modal commonality enhancement mechanism based on cross-attention with orthogonal constraints to improve semantic associations between modalities, ensuring coherent and aligned multi-modal features.", "weaknesses": "1. The design involves multiple modules, increasing the method's complexity in terms of reproducibility and spatio-temporal complexity compared to previous approaches.\n2. The experimental part of Label Dependency Analysis is not described in detail enough, for example, when the aligned sample pairs reach 80%, the effect of MCLEA is basically the same as the effect of TMEA on some indexes in the FB15K-DB15K dataset, but the text doesn't explain the reason for the phenomenon.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your valuable comments. We appreciate your recognition of our method's novelty, effectiveness, and the sufficiency of our experiments. In response to your concerns, we would like to address the following points:\n\n - **[W1: Reproducibility & Complexity]**:\nTo ensure reproducibility and promote research in this field, we plan to release our code. \nFor your concern about complexity, we would like to clarify that our method is not more complex than many previous approaches. \nMainstream methods for feature learning in MMKGs typically use different encoders to extract initial features and then design various fusion methods. \nDue to multiple modalities, these approaches can lead to slightly higher time and space complexity. \nHere, we calculate the time and space complexity of our method and one representative baseline, ACK-MMEA.\nWe denote the batch size as $B$ and the dimension size as $d$. \n    \n    The time complexity of ACK-MMEA consists of following parts: (1) Attribute Uniformization: This takes $O(|V|(d_T+d_E+d_I)d)≈O(|V|d^2)$. (2) Merge Operator: As it is a GAT structure, it spends $O(|V|d^2+|E|d)$. (3) Generate Operator: This involves average aggregation, so it takes $O(|E|d)$. (4) ConsistGNN: It is a 2-layer GCN structure, costing $O(2*3|E|d)$. (5) Joint Loss calculation: The entity similarity loss takes $O(3|V|d)$, the attribute similarity loss takes $O(2|V|d)$, and the neighbor dissimilarity takes $O(|E|d)$. \n    Thus, the total complexity is around $O(|V|d^2+(|V|+|E|)d)$. \n    \n    For our method, the time complexity consists of the following parts: (1) Linear Projections of the Pre-trained Visual and Attribute Features: They take $O(2Bd^2)$. (2) Loss for TransE: It takes $O(|T_{\\mathcal{R}}|  |T_{\\mathcal{R}}^{-}|d)$. (3) MMI: It first does relational projection for $O(2Bd^2)$, and then VAEs take $O(4Bd^2)$. Finally, the loss functions take $O(6Bd)$. (4) MCE: It consists of six multi-head cross-attention modules, taking $O(6\\eta B^2d)$. $L_{orth}$ further takes $O(6Bd)$. (5) Contrastive Learning Loss: It takes $O(4|H|d)$. Therefore, the total time complexity of TMEA is around $O(Bd^2+(B^2+|H|+|T_{\\mathcal{R}}|^2)d)$.\n\n    The space complexities of TMEA and ACK-MMEA are both $O(d^2 + |V|d)$.\n\n    Our method has a smaller time complexity than ACK-MMEA when the number of entities $|V|$ and edges $|E|$ is much larger than the batch size $B$, and when the term $B^2$ along with $|H|$ and $|T_{\\mathcal{R}}|^2$, is smaller than the sum of entities and edges. \n    This demonstrates the scalability of our method to large-scale and dense graphs, when $|V|$ and $|E|$ are large while we can keep $B$ moderate to maintain small complexity.\n    Regarding effectiveness, it is noted that our method outperforms ACK-MMEA by a large margin (0.867 vs 0.304).\n\n - **[W2: Phenomenon explanation]**: Thank you for your constructive advice. In Figure 4, there is a significant performance gap between our model and MCLEA, so we presume you are referring to MSNEA. When the aligned sample pairs reach 80%, even though their performance on the Hits@10 metric appears to be close, there is still a noticeable difference in MRR and Hits@1. This indicates that MSNEA performs poorly in aligning difficult samples.\n  We will add this content into the revised paper.\n\n - **[Q1: Modal missingness & MMI effect]**: For your concern about the meaning of considering modal missingness, we would like to clarify that KGs in the real world are generally incomplete but rather have missing information. We propose considering modality missingness to more robustly address this situation. The effect of MMI module is related to the extent of data missingness in the dataset. To validate the effectiveness of this module, we conducted ablation experiments as well as experiments with different proportions of missing modalities. As shown in Table 2, MMI shows an obvious improvement of ***nearly four percentage points in Hits@1*** on FB15K-YG15K dataset (w/o MMI: 77.9% vs TMEA: 81.8%). Additionally, as illustrated in Figure 3, our method demonstrates greater robustness: when only 20% of entities have the visual modality, TMEA achieves an MRR ***exceeding 0.5***, while the best baseline falls below 0.4. These results demonstrate the effectiveness of MMI module.\n\nWe hope these responses effectively address your concerns. We will make revisions to further clarify these aspects in our revised paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PWWaw4Rc0H", "reviewer": "Reviewer_fuge", "summary": "This paper aims to address the uncertainty in entity alignment within multimodal knowledge graphs (MMKGs). The authors design a MKE module to handle relations, attributes, and visual knowledge, enhancing attribute alignment and filtering through large language models and contextual learning. To address the issue of missing modalities, the paper introduces a MMI module that uses VAEs to generate pseudo-features. Additionally, MCE based on cross-attention and orthogonal constraints is developed to enhance semantic associations between different modalities.", "questions": "Why can the performance improve so significantly? What is your baseline, or is it entirely self-constructed?", "limitations": "See weakness and question", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper considers the issue of missing modalities in MMKGs, using VAEs to generate pseudo-features, achieving modality alignment in the latent space.\n\nIt systematically considers the relationships between relations, attributes, and images in MMKGs and achieves interactive enhancement.\n\nThe model diagram is concise and easy to understand.\n\nSOTA Experimental performance.", "weaknesses": "The problem description in the introduction is unclear, and the language logic is chaotic, failing to clearly explain the challenges faced by existing methods. The contribution points are also unclear, and the writing is poor.\n\nI understand that MMI is a relatively innovative part of the paper, but the ablation study shows its impact is minimal, indicating the limited effectiveness of the features supplemented by MMI.\n\nThe paper lacks innovation. Using large models for filtering and MMI is a rather straightforward approach. However, the large model is simply ChatGPT. Why didn't you consider fine-tuning other large models such as Llama?\n\nThe experimental table settings are problematic. For example, traditional methods like IPTransE have only 0.04 H1. Is it still necessary to compare with it? The method in this paper shows a significant improvement, far surpassing all baselines. For instance, on the DB15K dataset, the H1 performance of the previous EMNLP23's MoAlign method is **31.8**, and WWW23's method is **30.4**, while this paper achieves **86.7**. However, I find it hard to intuitively perceive the necessity for such performance improvement from the method presented in this paper. The MMI, which I consider innovative, shows a general impact in the ablation experiment.\n\nExperimental tables are missing. *Previous methods have tables set at 50% and 80%, but why are they **absent** here?* I couldn't find them in the appendix either. This paper lacks two major model performance tables.\n\nThe ablation study is not thorough or sufficient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your valuable comments. We appreciate your recognition of the soundness of our method, the clarity of the writing, and the SOTA performance. In response to your concerns, we would like to address the following points:\n\n\n - **[W1: Problem description & Challenges]**:\n    As described in our problem definition in Section 3,\n   we focus on addressing a fundamental problem in the field of KGs: multi-modal entity alignment.\n    We focus on tackling uncertain correspondences between inter-modal or intra-modal cues of entities for MMEA. It is a crucial issue that has not been solved in previous works and results in insufficient utilization of multi-modal knowledge, particularly in fusion and enhancement.\n  The challenges include three points: weak inter-modal associations, description diversity, and modality missing. \n  Please refer to Section 1 for more details about these challenges.\n  Our experimental results show promising performance, which proves that effectively addressing the problem of uncertain correspondences in MMEA task can lead to significant improvement.\n\n\n - **[W2: MMI effect]**:\n    We would like to clarify that the impact of MMI module is not minimal and its effect is related to the extent of data missingness in the dataset. \n    Its impact and effectiveness are demonstrated in both the ablation experiment and modality sensitivity experiment. As shown in Table 2, MMI shows an obvious improvement of ***nearly four percentage points in Hits@1*** \n    on FB15K-YG15K dataset (w/o MMI: 77.9% vs TMEA: 81.8%). In Figure 3, our method demonstrates greater robustness: when only 20% of entities have the visual modality, TMEA achieves an MRR ***exceeding 0.5***. \n\n\n - **[W1 & W3: Contributions & Innovation of LLM for filtering and MMI & Finetuned LLM]**:\n    For your concern about innovation, we would like to clarify that our approach is not a simple LLM filtering or MMI. \n    The technical innovations/contributions about them are as follows:\n     - To handle diverse attribute knowledge descriptions for attribute knowledge learning, our contribution is the design of a novel ***alignment-augmented abstract representation*** that incorporates the LLM and in-context learning into attribute alignment and filtering for ***generating and embedding the attribute abstract***.\n     - To mitigate the impact of modality absence, our contribution lies in the proposal to ***unify diverse modalities into a shared latent subspace and generate pseudo features*** via VAEs according to existing modal features in MMI module.\n\n    In addition to these two aspects of innovations, we specially design an ***inter-modal commonality enhancement mechanism*** based on cross-attention with ***orthogonal constraints*** to address the weak semantic associations in MCE module.\n\n    \n    Indeed, we have successfully addressed the critical issue of uncertain correspondences in MMEA by proposing a novel framework named TMEA, which significantly improves the performance of MMEA, a fundamental problem that benefits numerous downstream applications.\n\n    Regarding LLM selection, we would like to emphasize that our innovation does not lie in the use of LLMs and it lies in addressing the problem of uncertain correspondences by proposing a TMEA framework. \n    This framework can accommodate any LLM, and we have achieved promising results using a relatively straightforward LLM.\n\n\n - **[W4 & Q1: The reason for significant performance improvement & Baselines]**:\n    We would like to emphasize that the significant improvement in performance is attributed to our solution to the uncertain correspondence problem in the MMEA task, which enables more comprehensive feature learning of multi-modal knowledge.\n    Our primary innovation lies in the proposal of a holistic framework to tackle this important problem, and the experiments have demonstrated the overall effectiveness of both the framework and each individual component. As evidenced by the modality and component ablation studies in Table 2, the experimental results apparently suggest the improvement is the result of the combined contributions of all components.\n  \n    \n    Regarding baselines, we followed the standard experimental setup of the MMEA task like ACK-MMEA, MCLEA, and MSNEA, using well-adopted baselines that comprehensively cover traditional and multimodal entity alignment approaches. \n    Our results are either directly reproduced from public code or sourced from other papers, not self-constructed.\n\n    \n - **[W5: Missing 50% and 80% tables]**: These experimental results are presented in Figure 3, which show that TMEA provides superior performance with different proportions of training data.\n\n - **[W6: Ablation study]**:\n  The essential components in TMEA include MMI module (w/o MMI), MCE module (w/o MCE), alignment-augmented abstract representation (w/o AP), orthogonal constraint loss $L_o$ in MCE module (w/o $L_o$), MSE loss $L_{mse}$ in MMI module (w/o $L_{mse}$), and iterative strategy (w/o IT). \n    In Table 2, we have presented the ablation study including all these variants, and the results demonstrate the effectiveness of all essential components. If there are any further suggestions, we are open to supplementing the experiments.\n\nWe hope these responses effectively address your concerns. We will make revisions to further clarify these aspects in our revised paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper examines entity alignments in multi-modal knowledge graphs (MMKGs). It introduces a method called TMEA to address challenges in aligning inter-modal and intra-modal entity cues. The paper received mixed reviews.

After discussions with the SAC, we agreed that the authors presented a simple method yielding significant empirical improvements over the state-of-the-art. While the ablation performance  (removing the proposed MMI module within the pipeline) shows 4% contribution in terms of HITS@1, the other metrics improved like MR notably decreased from 32.9 to 26.3.  This suggests that the proposed module is effective for handling difficult samples. Although the novelty is somewhat limited, this contribution remains valuable to the community.

Weighing the pros and cons, we recommend accepting the paper as a poster presentation. However, the current version still requires substantial improvements. The authors should reorganize the paper to highlight their contributions and the importance of the performance improvements.
Decision: Accept (poster)