Multi-Modal Knowledge Graph Transformer Framework for Multi-Modal Entity Alignment
Authors: Qian Li
Cheng Ji
Shu Guo
Zhaoji Liang
Lihong Wang
Jianxin Li
Venue: EMNLP 2023 Findings (2023)
Keywords: Multi-Modal Entity Alignment
Multi-Modal Knowledge Graph
Transformer

Abstract: Multi-Modal Entity Alignment (MMEA) is a critical task that aims to identify equivalent entity pairs across multi-modal knowledge graphs (MMKGs). However, this task faces challenges due to the presence of different types of information, including neighboring entities, multi-modal attributes, and entity types. Directly incorporating the above information (e.g., concatenation or attention) can lead to an unaligned information space. To address these challenges, we propose a novel MMEA transformer, called Meaformer, that hierarchically introduces neighbor features, multi-modal attributes, and entity types to enhance the alignment task. Taking advantage of the transformer's ability to better integrate multiple information, we design a hierarchical modifiable self-attention block in a transformer encoder to preserve the unique semantics of different information. Furthermore, we design two entity-type prefix injection methods to redintegrate entity-type information using type prefixes, which help to restrict the global information of entities not present in the MMKGs.

Multi-Modal Knowledge Graph Transformer Framework for Multi-Modal Entity Alignment
Qian Li!.2, Cheng Ji Shu Guo?, Zhaoji Liang', Lihong Wang", Jianxin Li "School of Computer Science and Engineering, Beihang University, Beijing, China "Beijing Advanced Innovation Center for Big Data and Brain Computing, Beijing, China rean/Coordination CfnterofChina 4 School of Computer Science and Technology, University of Chinese Academy of Sciences [liqian, jicheng, 1ijx)@act. buaa. www..edu.com.. guoshuêcert. .n, liangzhaoji 23mmails ucas ucas.ac.c........oo..oooo. on,cn, wlh@isc. org.cn
National Computer Network EmtgerrEmmerennnyy Response ResponseTechhiica
Abstract
Multi-Modal Entity Alignment (MMEA) is critical task that aims to identify equivalent en- tity pairs across multi-modal knowledge graphs (MMKGs). However, this task faces challenges due to the presence of different types of infor- mation, including neighboring entities, multi- modal attributes, and entity types. Directly in- corporating the above information (e.g.. con- catenation attention) can lead unaligned information space. To address these challenges, we propose novel MMEA transformer, called MoAlign, that hierarchically introduces neigh- bor features, multi-modal attributes, and en- tity types enhance the alignment task. Tak- ing advantage of the transformer's ability to better integrate multiple information, we de- sign hierarchical modifiable self-attention block in a transformer encoder to preserve the unique semantics of different information. Furthermore, we design two entity-type prefix injection methods to integrate entity-type in- formation using type prefixes, which help to restrict the global information of entities not present in the MMKGs. Our extensive experi- ments benchmark datasets demonstrate that our approach outperforms strong competitors and achieves excellent entity alignment perfor- mance.
LarT
MIXI
MANGG 2
CVA
1
Figure I: Two examples of the MMEA task, where the entity pair in MMKG and MMKG 2 is entity seed and the pair in MMKG 1 and MMKG is not. The blue. green, and orange circles are entities, textual attributes and visual attributes.
MMEA (Liu et al., 2019; Li et al., 2023b; Liu et al., 2021; Lin et al., 2022) is challenging due to the heterogeneity of MMKGs (e.g.. different neigh- bors, multi-modal attributes, distinct types), which makes it difficult learn rich knowledge repre- sentations. Previous approaches such as PoE (Liu et al., 2019) concatenated all modality features to create composite entity representations but failed to capture interactions among heterogencous modal- ities. More recent works (Chen et al., 2020; Guo et al., 2021) designed multi-modal fusion mod- ules to better integrate attributes and entities, but still did not fully exploit the potential interactions among modalities. These methods also ignored inter-modality dependencies between entity pairs, which could lead to incorrect alignment. Gen- erally speaking, although MMKGs offer rich at- tributes and neighboring entities that could be use- ful for multi-mdoal entity alignment, current meth- ods have limitations in (i) ignoring the differen- tiation and personalization of the aggregation of heterogencous neighbors and modalities leading the misalignment of cross-modal semantics, and (ii) lacking the use of entity heterogeneity resulting in the -ondiiscrmiieeevee representations of different
1 Introduction
Multi-modal entity alignment (MMEA) is a chal- lenging task that aims to identify equivalent entity pairs across multiple knowledge graphs that fea- ture different modalities of attributes, such as text and images. To accomplish this task, sophisticated models are required to effectively leverage informa- tion from different modalities and accurately align entities. This task is essential for various applica- tions, such as cross- lingull information retrieval, question answering (Antol et al., 2015; Shih et al., 2016), and recommendation systems (Sun et al., 2020; Xu et al., 2021).
*Corresponding author.

---

meaning/types of entities.
unsuitable candidates and further enriches entity representations. To comprehensively evaluate the effectiveness of our proposed approach, we design training objectives for both entity and context eval- uation. Our extensive experiments on benchmark datasets demonstrate that our approach outperforms strong competitors and achieves excellent entity alignment performance. Our contributions can be summarized as follows.
Therefore, the major challenge of MMEA task is how to perform differentiated and personalized aggregation of heterogencous information of the neighbors, modalities, and types. Although such in- formation is beneficial entity alignment, directly fusing will lead to misalignment of the information space, as illustrated in Figure 1. Firstly, notable disparities between different modalities make di- rect alignment a challenging task. For example, both the visual attribute of entity Ruby in MMKG and the neighbor information of the entity Ruby in MMKG2 contain similar semantics of program- ming, but data heterogeneity may impede effective utilization of this information. Secondly, complex relationships between entities require a thorough understanding and modeling of contextual informa- tion and semantic associations. Entities such as the Ruby, the Perl, and the entity Larry Wall possess unique attributes, and their inter-relationshipss are non-trivial, necessitating accurate modeling based on contextual information and semantic associa- tions. Furthermore, the existence of multiple mean- ings for entities further exacerbates the challenge of distinguishing between two entities, such as in the case of the Ruby, which has different mean- ings in the MMKG1 and MMKG3 where it may be categorized as a jewelry entity or programming language entity, respectively.
We propose novel MMEA framework named MoAlign, which effectively integrates hetero- geneous information through the multi-modal KG Transformer.
We design a hierarchical modifiable self- attention block to build associations of task- related intra-modal features through the lay- ered introduction and design entity-type prefix to further enrich entity representations.
Experimental results indicate that the frame- work achieves state-of-the-art performance on the public multi-modal entity alignment datasets.
2 Preliminaries
Multi-Modal Knowledge Graph. A multi- modal knowledge graph (MKG) is represented by four sets: entities (E), relations (R), multi-modal attributes (AA), and triplets (7). The size of each set is denoted by Ne. NR. and NA. The multi-modal attributes are divided into text (ATT) and image (AI) attributes. The relation set R includes entity relations (RE), text attribute relations (RT), and image attribute relations (R.).. The set of triplets T includes entity triplets, text attribute triplets, and image attribute triplets.
To overcome the aforementioned challenges, we propose a novel Multi-Modal Entity Alignment Transformer named MoAlign'. Our framework hierarchically introduces neighbor, multimodal at- tribute, and entity types to enhance the alignment task. We leverage the transformer architecture, which is known for its ability to process heteroge- neous data, to handle this complex task. Moreover, to enable targeted learning on different modalities, we design a hierarchical modifiable self-attention block in the Transformer encoder, which builds associations of task-related intra-modal features through the layered introduction. Additionally, we introduce positional encoding to model entity rep- resentation from both structure and semanties si- multaneously. Furthermore, we integrate entity- type information using an entity-type prefix, which helps to restrict the global information of entities that are not present in the multi-modal knowledge graphs. This prefix enables better filtering out of
Multi-Modal Entity Alignment Task. Multi- modal entity alignment (Chen al., 2020; Guo et al., 2021: Liu et al., 2021: Chen et al., 2022) aims to determine if two entities from different multi- modal knowledge graphs refer to the same real- world entity. This involves calculating the similar- ity between pairs of entities, known as alignment seeds. The goal is to learn entity representations from two multi-modal knowledge graphs (M KG and M MKC (2) and calculate the similarity between a pair of entity alignment seeds (c, c') taken from these KGs. The set of entity alignment seeds is denoted as = {(e, e') e E, E E', e'}.
'The source code is available at ptttps///githut com/ xiaoqiant 9940510/ 19.4A1./Mo..ign.

---

MatMui
Feed Forward
Softmax
Hierarchical Modifiable Self-Attention Mechanism
Add Scale
Pi-Attention
Pi Pi-Attention
Pi Pi-Attention
Lx
MatMul
Inject
Inject
Matz
"Longuage"
Ruby hegremrie
M
Q
K
V
Hierarchical Modifiable Self-Attention Block
Prefix-Injected Self-Attention Mechanism
Modality
Structure Positional Encoding
Positional Encoding
Entity
Modifiable
Matrix
Bert
Input Embedding
Type Embedding
Text
Flatten Linear
+
Image
Type Prefix
MMKG
Embedding
Figure 2: The framework of the Multi Modal KG Transformer, MoAlign. The hierarchical modifiable self atf-atttetion block learns the entity by hierarchically integrating multi-modal attributes and neighbors. The prefix-injected self-attention mechanism introduces entity type information for alignment.
3 Framework
of word embeddings follows specific distribu- tion, whereas the distribution of image embeddings generated using this method is not consistent with the pretraining model's initial distribution. There- fore, to standardize the distribution of image em- beddings to be consistent with the distribution of the vocabulary used by the pretraining model, we perform the following operation:
This section introduces our proposed framework MoAlign. As shown in Figure 2, we introduce posi- tional encoding to simultancously model entity rep- resentation from both modality and structure. To hierarchically introduce neighbor and multi-modal attributes, we design hierarchical modifiable self- attention block. This block builds associations task-related intra-modal features through the layered introduction. Furthermore, for integrating entity-type information, we design prefix-injected self-attention mechanism, which helps to restrict the global information of entities not present in the MMKGs. Additionally, MoAlign also design train- ing objectives for both entity and context evaluation to comprehensively assess the effectiveness.
2y f -V, (elommmmmo..))/sdd(((( std( X, (1)
where mean(v.) and std(v,) are the mean and standard deviation of ve, respectively, and À is the standard deviation of truncated normal function.
3.1.2 Positional Encoding
For the transformer input, we additionally input two types positional encoding to maintain struc- ture information of multi-modal KG as follows.
3.1 Multi-Modal Input Embedding
3.1.1
Multi-Modality Initialization
Modality Positional Encoding. To enable the model to effectively distinguish between entities, textual attributes, image attributes, and introduced entity types, we incorporate a unique position code for each modality. These position codes are then passed through the encoding layers of the model to enable it to differentiate between different modali- ties more accurately and learn their respective fea- tures more effectively. Specifically, we assign po- sition codes 1, 2, 3, and 4 entities, textual attributes, image attributes, and introduced entity
The textual attribute is initialized by BERT (Devlin et al., 2019). For the visual attributes, to enable di- rect processing of images by a standard transformer, the image is split into sequence of patches (Doso- vitskiy et al., 2021). We then perform flatten oper- ation convert the matrix into one-dimensional vector similar to word embeddings, which is sim- ilar to embeddings BERT (Devlin al., 2019) and concatenate them to form the image embedding vector, denoted as v., However, the initialization

---

types, respectively. By incorporating this modality positional encoding information into the encoding process, our proposed model is able to incorporate mopality-specific features into the alignment task.
alignment task. The transformer encoder comprises three separate attention mechanisms, namely neigh- bor attention, textual attention, and visual attention. We utilize different sets of queries, keys, and values to learn diverse types of correlations between the input entity, its attributes, and neighbors.
Structure Positional Encoding. To capture the positional information of neighbor nodes, we intro- duce a structure positional encoding that assigns a unique position code to each neighbor. This allows the model to distinguish between them and effec- tively incorporate the structure information of the knowledge graph into alignment process. Specifi- cally, we assign position code of 1 to the current entity and its attributes. For first-order neighbors, we randomly initialize reference order and use it to assign position codes to each neighbor and its corresponding relation as 2n and 2n + 1, re- spectively. Additionally, we assign the same struc- ture positional encoding of attributes to their cor- responding entities. By doing so, the model can differentiate between different neighbor nodes and effectively capture their positional information.
Neighbor Multi-Head Attention. More specif- ically, in the first layer of the hierarchical block of the I-th transformer layer, employ the input entity as queries and its neighbors as keys and val- ues learn relations between the entity and its neighbors:
(1.1) All-Attm(e,,e,e)
(2)
where is the neighbor of entity and e!!1) is the representation of entity in the first layer.
Textual Multi-Head Attention. In the second layer of the hierarchical block, the input entity and its textual attributes are used as queries and keys/values to learn the correlations between the entity and its textual attributes.
To fully utilize the available information, we extract relation triplets and multi-modal attribute triplets for each entity. The entity is formed by combining multi-modal sequences as (ei, (en, (1)..,,.,.... Pm); er), where re), represents the i-th neighbor of entity e and its relation. (aj. represents the j-th attribute of entity e and its value Ug, and contains textual and visual attributes. n and m are the numbers of neighbors and attributes. er is the type embeddding.
(1.2) MHHAAtn(ee [i); [ag vil), (3)
where (at, 2x)) represents the textual attribute of entity and its value, and el..2) is the representation of entity e in the second layer.
Visual Multi-Head Attention. In the third layer of the hierarchical block, the input entity and its visual attributes are used similarly to the second layer, to learn the correlations between the entity and its visual attributes.
(L.3 MMH-Aunn(e [ays Dul, (ays vul), (4) where (a., U,) represents the visual attribute of entity e and its value, and el.33) is the representa- tion of entity in the third layer. By incorporating neighbor attention, textual attribute attention, and visual attribute attention, our model can capture various correlations between the input entity, its at- tributes, and neighbors in more effective manner.
3.2 Hierarchical Modifiable Self-Attention
To better capture the dependencies and relation- ships of entities and multi-modal attributes, we incorporate cross-modal alignment knowledge into the transformer. Specifically, a hierarchical mod- ifiable self-attention block is proposed to better capture complex interactions between modalities and contextual information. The block focuses on associations of inter-modal (between modalities, such as textual and visual features) by cross-modal attention. By doing so, the model can capture more informative and discriminative features.
3.2.2 Modifiable Self-Attention
To learn a specific attention matrix for building correlations among entities and attributes, we de- sign a modifiable self-attention mechanism. We manage to automatically and adaptively generate an information fusion mask matrix based on quence features. The length of the sequence and the types of information contains may vary de- pending on the entity, as some entities may lack certain attribute information and images.
3.2.1 Hierarchical Multi-Head Attention
We propose novel hierarchical block that incor- porates distinct attention mechanisms for different inputs, which facilitates selective attention toward various modalities based on their significance the

---

Modifiable Self-Attention Mechanism. To adapt the characteristics of different sequences, it is necessary to assign "labels" to various information when generating the sequence, such as using [F] to represent entities and [R] to represent relations. This way, the positions of various information in the sequence can be generated, and the mask matrix can be generated accordingly.
Prefix-Injected Self-Attention Mechanism. Specifically, we create two sets of prefix vectors for entities, p", p" E R for keys and values separately, where ne the number of entity types. These prefix vectors are then concatenated with the original key K and value V. The multi-head attention in Eq.(2) is performed on the newly formed prefixed keys and values follows:
These labels need to be processed by the model's tokenizer after inputring the model and generating the mask matrix to avoid affecting the subsequent generation of embeddings. We can still modify the vocabulary to allow the tokenizer recognize these words and remove them. The mask matrix can be generated based on the positions of various information, represented by labels, as follows:
AH--Attl Q)
-.hx)
h h,=Pi--tthe QW? [KW p,,,
where dy is typically set to d/N, and N is number of head. h, denotes i-th output of self-attention in I-th layer. W. E Idxd W W W Rdxdy learnable parameters. In this paper, we use the type embedding eT as the prefix.
In order to effectively incorporate alignment in- formation, we utilize the mask token influence at- tention weights. The mask token serves as a means to capture effective neighbors and reduce computa- tional complexity. To achieve this, we set attention weights to a large negative value based on the mask matrix M,j in the modifiable self-attention mechanism. Consequently, atlention weights for the mask token are modified as follows:
1, i((i,),)ETT or T()) T())
Mij- 0, otherwise
(5)
where T() is the type mapping function of entities and attributes.
Subsequently, residual connections and layer normalization are utilized to merge the output of hierarchical modifiable self-attention e' and apply position-wif wise feed-forward network to each ele- ment in the output sequence of the self-attention. The output sequence, residual connections, and layer normalization are employed as:
QK?
Pi- TT-Attn(Q, Vi
V.. (8)
vdz
where Q, K,,, and V, represent the query of the entity, the key containing entity type. and the value containing entity type matrices, respectively. de refers the dimensionality of key vectors, while m represents the attention weights.
e -LayerNorm (e ((-D++FN((e")) (6)
where el is the output of the hierarchical modifiable self-attention in the 1-th layer of the transformer. The use of residual connections and layer normal- ization helps stabilize training and improve perfor- mance. It enables our model to better understand and attend different modalities, leading more accurate alignment results.
Prefix-Injected Feed Forward Network. Re- cent research suggests that the feed-forward layers within the transformer architecture store factual knowledge and can be regarded as unnormalized key-value memories (Yao et al., 2022). Inspired by this, we attempt inject entity type into each ferw-f layer to enhance the model's ability to capture the specific information related to the en- tity type. Similar to prefix injection in attention, we first repeat the entity type to create two sets of vectors, th "" E Rd These vectors are then concatenated with the original parameter matrices of the first and second linear layers.
3.3 Entity-Type Prefix Injection
Prefix methods can help improve alignment accu- racy by providing targeted prompts to the model to improve generalization, including entity type infor- mation (Liu et al., 2023). The entity-type injection introduces a type of information that takes in- put pair of aligned seed entities from different MMKGs and uses entities as prompts. The goal to inject entity-type information into the multi-head attention and feed-forward neural networks.
FFN(E) - [ Φ Φ1) (Wy; T").
(9) where denotes the non-linearity function, E rep- resents the output of the hierarchical modifiable self-attention. W wy are the parameters.

---

3.4 Training Objective
4.2 Comparision Methods
order to effectively evaluate the MMEA from multiple perspectives, we propose a training objec- tive function that incorporates both aligned entity similarity and context similarity. This objective function serves to assess the quality of entity align- ment and takes into consideration both the entities themselves and their surrounding context.
We compare our method with three EA baselines (TransE (Bordes et al., 2013), GCN-align (Wang et al., 2018a), and AttrGNN (Liu et al., 2020b)), which aggregate text attributes and relation infor- mation, and introduced image attributes initialized by VGGI6 for entity representation using the same aggregation method as for text attributes. We fur- ther use three transformer models (BERT (De- vlin et al., 2019), VIT (Dosovitskiy et al., 2021), and CLIP (Radford et al., 2021)) to incorporate multi- mudti-modal information, and three MMEA meth- ods (PoE (Liu et al., 2019), Chen et al. (Chen et al., 2020), HEA (Guo al., 2021), EVA (Liu et al., 2021). and ACK-MMEA (Li et al., 2023a)) to focusing on utilizing multi-moda attributes. The detail is given in Appendix B.
Aligned Entity Similarity. The aligned entity similarity constraint loss is designed to measure similarity between aligned entity pairs and ensure that embeddings of these entities are close to each other in the embedding space.
LEA = -im(e,, e') sim(e, sim)-simme, (10)
where (e, represent the final embeddings of the aligned seed entities ee,e') from knowledge graphs KG1 and KG2. e and e' denote the nega- tive samples of the seed entities. sim(--- refers to the cosine distance between the embeddings.
4.3 Implementation Details
We utilized the optimal hyperparameters reported in the literature for all baseline models. Our model was implemented using PyTorch, an open- source deep learning framework. We initialized text and image attributes using bert-base-uncased and VGG16*, respectively. To ensure fairness, all baselines were trained on the same data set parti- tion. The best random dropping rate is 0.35, and coefficients a, were set to 5 and respectively. All hyperparameters were tuned on validation data using trials. All experiments were performed on a server with one GPU (Tesla V100).
Context Similarity. The context similarity loss is employed to ensure that the context representations of aligned entities are close to each other in the embedding space.
LCon ssimssmm(opp (:sim((."))ssmmo..)))) oiiiooo"'sima.ma.. o'), (11)
where (o, o') denote the final context representa- tions ([cls] embedding) of the aligned seed entities (e, e'). and o' represent the negative samples of the seed entities' context.
The total alignment loss computed as:
4.4
Main Results
= aLEA + CCon
(12)
To verify the effectiveness of MoAlign, we report overall average results in Table 1. shows per- formance comparisons on both two datasets with different splits on training/testing data of alignment seeds, i.e., 2:8, 5:5, and 8:2. From the table, we can observe that: 1) Our model outperforms all base- lines of both EA, multi-modal Transformer-batt based, and MMEA methods, in terms of three metries both datasets. It demonstrates that our model is robust to different proportions of training resources and learns good performance on few-shot data. 2) Compared to EA baselines (1-3), especially for MRR and Hits 1, our model improves 5% and 9% up on average on FB15K-DBISH 15K and FB 15K- YAGO15K, tending to achieve more significant improvements. It demonstrates that the effective-
where a, B are learnable syper--arameters.
4
Experiment
4.1
Dataset
We have conducted experiments two of the most popular datasets, namely 1B15K-DB15K and FB15K-YAGO1S as described in (Liu et al., 2019). The FBISK-DBISK dataset2 is an en- tity alignment dataset of FB15K and DB15K MMKGs, while the latter is a dataset of FB15K and YAGOI5K MMKGs. Consistent with prior works (Chen et al., 2020; Guo et al., 2021), we split each dataset into training and testing sets in proportions of 2:8, 5:5, and 8:2, respectivcly. The MRR, Hits @1 1, and Hits @ 10 are reported for eval-
uation on different proportions of alignment seeds.
#htpss//githha.ccommmmiprrrmmmm

---

YBISK-DBISK (20%)
1BISK-DB1SK (80%)
FBI5K DB15K (80%)
Methods
His@1
His@ 1D
Hise
His@@1
Hise@10
TrandE (Bondes 2013)
2013)
13.4
7.8
24.0
30.6
23.0
44.6
50.7
42.6
65.9
GCN-align (Wing sl., 2018a)
8.7
5.3
17.4
29.3
22.6
43.5
47.2
41.4
63.5
AMtrGNN (Liu et 2020b)
34.3
25.2
53.5
54.7
47.3
72.1
70.3
67.1
83,9
BERT (Devlin et al., 2019)
32.6
24.3
48.0
49.6
45.2
67.9
65.3
64.5
80.1
(Dosovitskly at. 2021)
25.1
45.5
69.0
85.7
CLIP (Radford et al., 2021)
35.4
27.0
55.3
$4.1
48.7
71.4
73.9
68.3
86.0
al., 2019)
25.1
46.4
72.1
82.0
Chen al. (Chon et 2020)
35.7
26.5
54.1
51.2
41.7
70.3
68.5
59,0
86,9
HEA (Guo et 2021)
12.7
36.9
26.2
58.1
41.7
78.6
EVA (L.u 2021)
35.2
28.9
54.5
53.8
45.3
72.9
71.6
63.5
85.1
ACK-MMRA (1. et al., 2023aa
38.7
30.4
34.9
62.4
56.0
23.6
25.2
68.2
87.4
MoAliga (ours)
(11.4)
(-.1)
(3.4
(7.6
(4.33
(1.3))
69.9 (11.6)
(10.8)
7B15K-YAG015K (20%)
7BI5K-YAGOIST (50%)
TB15K-YYG0155 (80%)
Methods
MRR
Hiis@1
Hiss
MRR
His@
Hiss 10
MRR
Hiis@
Hiss@ 10
Transk (Borks al.,
6.4
20.3
GCN-align (Wing sl., 20188)
15.3
8.1
23.5
29.4
23.5
42.4
47.7
40.6
64.3
AteeNN al., 2020h)
78.7
BERT (Devlin al., 2019)
30.5
23.6
39.0
48.7
43.1
62.4
67.3
60.8
81.2
(Dosovitskly at.
32.4
26.8
44.9
$2.0
45.7
67.5
71.3
63.1
83.0
CLIP (Radford 202.1)
34.8
29.3
47.1
56.8
49.0
20.2
72.1
65.2
85.2
Pok (Liu et al., 2019)
15.4
11.3
22.9
414
34.7
$3.6
63.5
57.3
74.6
2020)
IIEA (Guo et ol., 2021)
10.5
31.3
26.5
58.1
43,3
80.1
2021)
ACK-MMEA (Li et 2023g)
36.0
28.9
49.6
59.3
53.5
69.9
74.4
67.6
8644
MoAAign (ours)
(018)
(10.3)
$2.5
(*2.9)
61.7 (12.4)
(51.5
(1.3
76.9 (?2.5)
68.9 (11.3)
$8.4 (?2.0)
Table 1: Main experiments on FB15K-DB15K (top) (%) and FB15K- -KAAGG015K (bottom) (%) with different proportions of MMEA seeds. The best results are highlighted in bold, and underlined values are the second best result. "1" means the improvement compared to the second best result, and ". means that results are not available.
FB15K-DB15K (80%)
FB15K-DB15K (80%)
Variants
MRR
Hits@1
Avg
MRR
His@
Hise 10
4 Avg
Mogl (ours)
77.3
69.9
88.2
76,9
68.9
88.4
w/o modifiable layer
75.8
66.4
85.9
12.4
74.2
65.3
85.7
13.0
w/o modifiable self-attention
76.5
67.1
86.4
11.8
74.6
66.7
86.2
12.2
repl. multi-head self-attention
12.6
w/o information
76.2
67.8
87.3
11.4
74.1
66.5
86.7
12.3
w/o context loss
76.9
68.2
87.1
11.1
75.4
67,2
86.3
J1.8
w/o text attribute image attribute
75.1
65.3
85.7
13.1
73.4
65.0
86.4
13.1
86.0
12.7
74.0
66.8
85.4
12.7
Table 2: Variant experiments 1B15KDDB15K (80%) and FBISK-YOGAISK (80%). "w/o" means removing corresponding module from complete model. "repl." means replacing corresponding module with other module.
ness of multi-modal context information benefits incorporating alignment knowledge. 3) Compared transformer-basee baselines, our model achieves better results and the transformer- based baselines perform better than EA baselines. It demonstrates that transformer- based structures can learn better MMEA knowledge. 4) Compared to MMEA baselines, model designs Hierarchi- cal Block and modifiable self-attention mechanism, the average gains of our model regarding MRR, @1t 1, and Hits@ 10 are 2%, 1.4%, and 1.7%, respectively. The reason is that our method incor- porates multi-modal attributes and robust context entity information. All the observations demon- strate the effectiveness of MoAlign.
4.5 Discussions for Model Variants
To investigate the effectiveness of each module in MoAlign, we conduct variant experiments, show- casing the results in Table The "L" means the value of performance degradation compared to the MoAlign. We can observe that: 1) The impact of the Hierarchical Block tends to be more significant. We believe the reason is that the adaptive intro- duction of multi-modal attributes and neighbors captures more clucs. 2) By replacing the modifi- able self-attention the multi-head self-attention, the performance decreased significantly. demon- strates that the modifiable self-attention captures more effective modti-modal attribute and relation information. 3) When we remove all image at-
experiai
to

---

78.0
Del. text attributes
Intert. Textual Attributes
Intert. Neighbors Intert. Random
80
Del. image attributes
attributes
77.5
2002
Interf. Visual Attributes
77.0
MRR (%)
76.5
70
76.0
75.5
65
60
74.5
74.0
AttrGNN
CLIP
PoE Chan Models
ACK
5%
10%
15%
20%
25%
30%
Interference Proportion TST-MMEA
Figure 3: Results of deleting attributes FBISK- DB15K (80%). "Del." means deleting the correspond- ing attribute.
Figure 4: Results of interference attributes neighbors on FB 155--BBBISS 15KK (80%). "Interf." means interference some proportion of attributes or neighbors.
FBISK-DBI5K (80 %)
Variants
MRR
Hits@1
Hits@ 10
A Avg
tual attributes, and visual attributes. As shown in the Table 3, the introduction order has a signifi- cant impact on our model. Specifically, the best performance is achieved when textual attributes, visual attributes, and neighbor information are in- troduced in that order. This suggests that aggre- gating attribute information to learn good entity representation first, and then incorporating neigh- bor information, can effectively utilize both the attributes and neighborhood information of nodes.
MoAlign (N - - T)
77.3
69.9
88.2
MoAlign (N + T V)
76.3
68.0
87.5
11.2
MoAlign (V +N -->17
76.0
67.4
86.4
11.9
MoAlign (V + T N)
75.6
67.3
12.1
MoAlign (T -NN V)
75.8
67.7
85.9
12.0
MoAlign (T - V N)
76.0
66.9
86.1
12.1
Table 3: Order Impact on FBISK-DBI5K (80%).
tributes as "w/o image attribute", our method drops 2.7% and 2.7% on average on FB15K-DB 55KKB1 and FB15K-YAGE YAGO15K. It demonstrates that image at- tributes improve model performance and our method utilizes image attributes effectively by cap- turing more alignment knowledge.
4.7 Impact of Interference Data
We investigate the impact of interference data on our proposed method for MMEA. Specifically, we randomly replace 5%, 10%, 15%, 20%, 25% and 30% of the neighbor entities and attribute informa- tion to test the robustness of our method, shown in Figure 4. The experimental results demonstrate that our method exhibits better tolerance to inter- ference compared to the baseline methods. This suggests that our approach, which incorporates hi- crarchical information from different modalities and introduces type prefix to entity representations, is capable of handling interference data and im- proving the robustness of the model.
4.6 Impact of Multi-modal Attributes
To further investigate the impact of multi-modal attributes on all compared methods, we report the results by deleting different modalities of attributes, as shown in Figure 3. From the figure, we can ob- serve that: 1) The variants without the text or im- age attributes significantly decline on all evaluation metrics, which demonstrates that the multi modal attributes are necessary and effective for MMEA. 2) Compared other baselines, our model derives bet- ter results both the case of using all multi-modal attributes or abandoning some of them. It demon- strates our model makes full use of existing multi- modal attributes, and modtll attributes are effective for MMEA. All the observations demon- strate that the effectiveness of the MMKG trans- former encoder and the type-prompt encoder.
Conclusion
This paper proposes a novel MMEA framework. It incorporates cross moosl alignment knowledge using a two-stage transformer encoder to better capture complex inter-modality dependencies and semantic relationships. It includes a MMKG trans- former encoder that uses self-attention mechanisms to establish associations between intra-modal fea- tures relevant to the task. Our experiments show that our approach outperforms competitors.
In addition, investigate the impact of the order in which multi-modal attributes are introduced to the model, we conduct experiments with different orders of introducing neighbor information, tex-

---

Limitations
Acknowledgment
Our work hierarchically introduces neighbor, mul- timodal attribute, and entity types to enhance the alignment task. Empirical experiments demon- strate that our method effectively integrates het- erogeneous information through the multi-modal KG Transformer. However, there are still some limitations of our approach can be summarized as follows:
We thank the anonymous reviewers for their in- sightful comments and suggestions. Jianxin Li is the corresponding author. The authors of this pa- per were supported by the NSFC through grant No.62225202 62106059.
References
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In ICCV, pages 2425-2433.
Due to the limitation of the existing MMEA datasets, we only experiment on entity, text, and image modalities to explore the influence of multi-modal features. We will study more modalities in future work.
Antoine Bordes, Nicolas Usunier, Alberto Gareía- Durán, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi- relational data. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Pro- ceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pages 2787-2795.
Our approach employs the transformer encoder architecture, which entails a substamtial time over- head. In forthcoming investigations, we intend to explore the feasibility of leveraging prompt- based techniques to mitigate the computational burden and expedite model training.
Xianshuai Cao. Yuliang Shi, Jihu Wang, Han Yu, Xin- jun Wang, and Zhongmin Yan. 2022. Cross modal knowledge graph contrastive learning for machine learning method recommendation. In MM '22: The 30th ACM International Conference on Multimedia, Lisbou, Portugal, October 10 14, 2022, pages 3694- 3702, ACM.
Ethics Statement
In this work, we propose new MMEA frame- work that hierarchically introduces neighbor, multi- modal attribute, and entity types to benchmark our architecture with baseline architectures on the two MNEA datasets.
Liyi Chen, Zhi Li, Yijun Wang, Tong Xu, Zhefeng Wang, and Enhong Chen. 2020. MMEA: entity align- ment for multi-modal knowledge graph. In Knowl- edge Science, Engineering and Management 13th International Conference, KSEM 2020, Hangzhou, China, August 28-30, 2020, Proceedings, Part I, vol- ume 12274 of Lecture Notes in Computer Science. pages 134-147. Springer.
Data Bias. Our framework is tailored for multi- modal entity alignment in the general domain. Nonetheless, its efficacy may be compromised when confronted with datasets exhibiting dissimilar distributions or in novel domains, potentially lead- ing to biased outcomes. The experimental results presented in the section are predicated on particular benchmark datasets, which are susceptible to such biases. As result, it is imperative exercise cau- tion when assessing the model's generalizability and fairness.
Zhuo Chen, Jiaoyan Chen, Wen Zhang, Lingbing Guo. Yin Fang. Yufeng Huang, Yuxia Geng, Jeff Z. Pan, Wenting Song, and Huajun Chen. 2022. Meaformer: Multi-modal entity aligmment transformer for meta modality hybrid. CORR, abs/2212.14454.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Computing Cost/Emission. Our study, which in- volves the utilization of large-scale language mod- els, entails a substantial computational overhead. We acknowledge that this computational burden has a detrimental envirommental impact in terms of carbon emissions. Specifically, our research neces- sitated a cumulative 588 GPU hours of computation utilizing Tesla V100 GPUs The total carbon foot- print resulting from this computational process is estimated to be 65.27 kg of CO2 per rum, with a total of two runs being conducted.
Alexey Dosovitskiy, Lucas Beyer. Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x 16 words: Transformers for image

---

recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net,
Zhenxi Lin, Ziheng Zhang, Meng Wang. Yinghui Shi, Xian Wu, and Yefeng Zheng. 2022. Multi-modal con- trastive representation learning for entity alignment. In Proceedings of the 29th International Confer- ence on Computational Linguistics, COLING 2022, Gyeongju, Republic of Korea, October 12-17, 2022, pages 2572-2584, International Committee on Com- putational Linguistics.
Xiang Fang, Daizong Liu, Pan Zhou, and Yuchong Hu, 2022. Multi-modi cross-domain alignment network for video moment retrieval. IEEE Transactions on Mullimedia.
Fangyu Liu, Muhao Chen, Dan Roth, and Nigel Collier. 2021, Visual pivoting for (unsupervised) entity align- ment. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-This Third Conference Innovative Applications of Artificial Intelligence, IAAI 2021. The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Vir- tual Event, February 2-9, 2021, pages 4257-4266.
Xiang Fang. Daizong Liu, Pan Zhou, and Guoshun Nan. 2023a. You can ground earlier than see: An effective and efficient pipeline for temporal sentence ground- ing in compressed videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 2448-2468 2460.
Xiang Fang, Daizong Liu, Pan Zhou, Zichuan Xu, and Ruixuan Li. 2023b. Hierarchical local-global trans- former for temporal sentence grounding. IEEE Trans- actions on Multimedia.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig, 2023. Pre- train, prompt. and predict: A systematic survey of prompting methods in natural language processing. ACM Comput. Surv.
Hao Guo, Jiuyang Tang. Weixin Zeng, Xiang Zhao, and Li Liu. 2021. Multi-modal entity alignment in hyperbolic space. Neurocomputingg 661-599-6077
Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang. Qi Ju, Haotang Deng, and Ping Wang. 2020a. K-BERT: en- abling language representation with knowledge graph. In The Thirty-Fourdl AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innova- rive Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educa- tional Advances in Artificial Intelligence, EAAT 2020, New York, NY, USA, February 7-12, 2020, pages AAAI Press.
Phillip Howard, Arden Ma, Vasudev Lal, Ana Paula Simöes, Daniel Korat, Oren Pereg, Moshe Wasserblat, and Gadi Singer. 2022. Cross-domain aspect extraction using transformers augmented with knowledge graphs. In Proceedings of the 31tt ACM International Conference on Information & Knowl- edge Management, Atlanta, GA, USA, October 17-21, 2022, pages 780-790. ACM.
Zhiwei Hu, Victor Butiérrez-asullo,,, Zhiliang Xiang, Ru Li, and Jeff Z. Pan. 2022. Transformer-based entity typing in knowledge graphs. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 5988-6001. Association for Computational Linguistics.
Xiao Liu, Shiyu Zhao, Kai Su, Yukuo Cen, Jiezhong Qiu, Mengdi Zhang, Wei Wu, Yuxiao Dong, and Jie Tang. 2022. Mask and reason: Pre-training knowl- edge graph transformers for complex logical queries. In KDD '2: The 28th ACM SIGKDD Conference Knowledge Discovery and Data Mining. Washington, DC, USA, August 14- - 18, 2022, pages 1 1120-112 130. ACM.
Ye Liu, Hui Li, Alberto Carcía-Durán, Mathias Niepert, Daniel Fhooo-RRuii and David S. Rosenblum. 2019. MMKG: multi-modal knowledge graphs. In The Se- mantic Web 16th International Conference, ESWC 2019, Portoroč, Slovenia, June 2-6, 2019, Proceed- ings. volume 1 15503 of Lecture Notes in Computer Science, pages 459-474. Springer.
Jin Jiang, Mohan Li, and Zhaoquan Gu. 2021. A sur- vey on translating embedding based entity alignment knowledge graphs. In Sixth IEFE International Conference on Data Science in Cyberspace, DSC 2021, Shenzhen, China, October 9-11, 2021, pages 187-194, IEEE
Qian Li, Shu Guo, Yangyifei Luo, Cheng Ji. Li- hong Wang, Jiawei Sheng, and Jianxin L.i 2023a. Attribute-consisteen knowledge graph representation learning for moali entity alignment. In Pro- ceedings of the ACM Web Conference 2023, WWW 2023, Austin, TX. USA, 30 April 2023 4 May 2023. pages 2509. ACM.
Zhiyuan Liu, Yixin Cao, Liangming Pan, Juanzi Li, and Tat- Tat-Seng Chua. 2020b. Exploring and evaluating attributes, values, and structures for entity alignment. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. EMNLP 2020, Online, November 16-20, 2020, pages 6355- 6364. Association for Computational Linguistics.
Yangning Li. Jiaoyan Chen, Yinghui Li. Yuejia Xi- ang, Xi Chen, and Haitao Zheng. 2023h. Vision, deduction and alignment: An empirical study on multi- modal knowledge graph alignment. CoRR, abs/2302.08774.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language

---

supervision. In Proceedings the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8748-8763. PMLR.
modii knowledge graph for ete-streaming e- commerce. CIKM '21: The 30th ACM Interna- rional Conference on Information and Knowledge Management, Virtual Event, Queensland, Australia, November 1
5, 2021, pages
ACM.
Kevin J Shih, Saurahh Singh, and Derek Hoiem. 2016. Where to look: Focus regions for visual question answering. In CVPR, pages 4613-4621.
Sen Yang, Xuanhan Wang, Lianli Gao, and Jingkuan Song. 2022. MKE-GGN: multi-modal knowledge embedded graph convolutional network for skeleton- based action recognition in the wild. In IEEE Inter- national Conference Multimedia and Expo, ICME 2022, Taipei, Taiwan, July 18-22, 2022, pages 1-6. IEEE.
Rui Sun, Xuezhi Cao, Yan 7hao, Junchen Wan, Kun Zhou, Fuzheng Zhang. Zhongyuan Wang, and Kai Zheng. 2020. Multi-modal knowledge graphs for recommender systems. In CIKM '20: The 29th ACM International Conference on Information and Knowl- edge Management, Virtual Event, Ireland, October 19-23, 2020, pages 1405-1414. ACM.
Yunzhi Yao, Shaohan Huang, Li Dong, Furu Wei, Huajun Chen, and Ningyu Zhang. 2022. Kformer: Knowledge injection transformer feed-forward layers. In Natural Language Processing and Chi- nese Computing: 11th CCF International Confer- ence, NLPCC 2022, Guilin, China, September 24-25, 2022, Proceedings, Part I, pages 131-143. Springer.
Jiaan Wang. Fandong Meng, Duo Zheng, Yunlong Liang, Zhixu Li, Jianfeng Qu, and Zhou. 2022. survey on cross-lingual summarization. Trans. Assoc. Comput. Linguistics, 10:1304-1322
Xiangru Zhu, Zhixu Li, Xiaodan Wang. Xueyao Jiang, Penglei Sun, Xuwu Wang. Yanghua Xiao, and Nicholas Jing Yuan. 2022. Multi-modal knowledge graph construction and application: A survey. CoRR, abs/2202.05786.
Jiaan Wang. Fandong Meng, Duo Zheng, Yunlong Liang, Zhixu Li. Jianfeng Qu, and Jie Zhou. 2023, Towards unifying multi Manti-linggal and cross- loogslln summarization. In Proceedings of the 61st Annual Meeting of the Association for Computational Lin- guistics (Volame Long Papers). ACL 2023, Toronto. Canada, July 9-14, 2023, pages 15127-15143, Asso- ciation for Computational Linguistics.
Xiang Wang, Xiangnan He, Yixin Cao, Meng Liu, and Tat-Seng Chua. 2019. KGAT: knowledge graph at- tention network for recommendation. In Proceedings of the 25th ACM SICKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019, pages 950- 958. ACM.
Zeshi Wang, Mohan Li, and Zhaoquan Gu. 2021. review of entity alignment based on graph convolu- tional neural network. In Sixth IEEE International Conference on Data Science in Cyberspace, DSC 2021, Shenzhen, China, October 9-11, 2021, pages 144-151 IEEE
Zhichun Wang, Qingsong Lv, Xiaohan Lan, and Yu Zhang, 2018a. Cross-lingual knowledge graph alignment via graph convolutional networks. In Pro- ceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, Brussels, Bel- gium, October 31- November 4. 2018, pages 349- 357.
Zhichun Wang. Qingsong Lv, Xiaohan Lan, and Yu Zhang. 2018b. Cross-lingual knowledge graph alignment via graph convolutional networks. In Pro- ceedings of the 2018 conference on empirical meth- ods natural language processing, pages 349-357.
Guohai Xu, Hehong Chen, Feng-Lin Li, Fu Sun, Yun- zhou Shi, Zhixiong Zeng, Wei Zhou, Zhongzhou Zhao, and Ji Zhang. 2021. Alime MKG: A

---

A Related Work
uses a graph attention mechanism capture the complex relations between entities in a knowledge graph, and a Transformer to learn representations of the entities for downstream tasks such as link pre- diction and entity recommendation. The K-BERT model (Liu et al., 2020a) extends this approach by pre-training Transformer tual data, and then fine-tuning it on knowledge graph to improve entity and relation extraction. The transformer model has the ability model long- range dependencies, where entities and their asso- ciated modalities can be distant from each other in the knowledge graph. Furthermore, Transformers utilize attention mechanisms to weigh the impor- tance of different inputs and focus on relevant in- formation, which is particularly useful for aligning entities across different modalities.
A.1 Multi-Modal Entity Alignment
In the real-world, due to the multi-modal nature of KGs, there have been several works (Zhu et al.. 2022; Wang et al., 2021; Jiang et al., 2021; Fang et al., 2022) that have started to focus on MMEA technology. One popular approach is to use em- beddings to represent entities and their associated modalities (Cao et al., 2022; Yang et al., 2022). These embeddings can then be used to measure the similarity between entities and align them across different KGs. Wang et al. (Wang et al., 2018b) proposed framework that uses cross-modal em- beddings align entities across different modal- ities, such as text and images. The model maps entities from different modalitics into a shared em- bedding space, where entities that correspond to the same real-world object are close to each other. However, this approach cannot capture the poten- tial interactions among heterogeneous modalities, limiting its capacity for performing accurate entity alignments. To address this limitation, some re- searchers have proposed mudti-modal knowledge embedding methods that can discriminatively gen- erate knowledge representations of different types of knowledge and then integrate them. Chen et al. (2020) proposed a method that uses multi-modal fusion module to integrate knowledge representa- tions of different types. Similarly, Guo et al. (2021) proposed GNN-based model that learns to ag- gregate information from different modalities and propagates across the knowledge graph align entitics. developed a hyperbolic
large corpus of tex-
B Comparision Methods
We compared our method with three EA baselines that aggregate text attributes and relation informa- tion, and introduced image attributes initialized by VGG16 for entity representation using the same aggregation method as for text attributes. The three EA methods compared are as follows: (1) TransE (Bordes al., 2013) assumes that the entity embedding should be close to the attribute embed- ding a plus their relation r. (2) GCN-align (Wang et al., 201 8a) transfers entities and attributes from each language to a common representation space through GCN. (3) AttrGNN (Liu et al., 2020b) di- vides the KG into multiple subgraphs, effectively modeling various types of attributes.
en- tity aligmment (HEA) approach that combines both attribute and entity representations in the hyper- bolic space and uses aggregated embeddings to pre- dict alignments. EVA (Liu et al., 2021) combines images, structures, relations, and attributes infor- mation for the MMEA with a learnable weighted attention to learn the importance of each modal at- tributes. Despite these advances, existing methods often ignore contextual gaps between entity pairs, which may limit the effectiveness of alignment.
We also compared our method with three trans- former models that incorporate multi-modal in- formation: (4) BERT (Devlin et al., 2019) is a pre-trained model to generate representations. (5) VIT (Dosovitskiy et al., 2021) is visual trans- former model that partitions an image into patches and feeds them input sequences. (6) CLIP (Rad- ford et al., 2021) is joint language and vision model architecture that employs contrastive learn- ing.
In addition, we compared our method with four MMEA methods focusing on utilizing multi-modal attributes: (7) PoE (Liu et al., 2019) utilizes im- age features and measures credibility by matching the semantics of entities.
A.2 Knowledge Graph Transformer
The Transformer architecture, originally proposed for natural language processing tasks, has been ap- plied various knowledge graph tasks well (Liu et al., 2022; Hu et al., 2022; Howard et al., 2022; Wang et al., 2023, 2022; Fang et al., 2023b,a). For example, the KGAT model (Wang et al., 2019)
(8) Chen et al. (Chen et al., 2020) designs fusion module to integrate multi-modal attributes. (9) HEA (Guo et al., 2021) characterizes MMKG in hyperbolic space. (10)

---

EVA (Liu al., 2021) combines multi-modal at- tributes and relations with an attention mechanism to learn the importance of modality. (11) ACK- MMEA (Liu et al., 2021) designs an attribute- consistent KG representation framework to com- pensate contextual gaps.


Reviews:
[{"review_id": "VHmWQQLCvH", "reviewer": "Reviewer_h5aT", "summary": "", "questions": "1. How is the problem addressed in this paper different from others?\n\n2. Why does “N-V-T” work better? Is it related to the dataset?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** More clearly description of “incorporating/aggregating the different types of information”. \\\n**A1:** Thanks for the encouraging comments. We design the hierarchical modifiable self-attention for the problem of aggregating different types of information. Specifically, we focus on the information of neighbors, multi-modal attributes, and types, which are the major features and structural information used in most works. However, directly aggregating such different information leads to a misaligned information space, as they provide multiple perspectives on entity cognition. The information of neighbors contains the context information of the target entity. The multi-modal attributes incorporate the features unique to the entity itself, which are the internal characteristics of the target entity. The entity type constrains the possible scope of the entity. Different from the previous works, we design a new self-attention mechanism to hierarchically aggregate different information. We will add the description in Section 1.\n\n**Q2:** Make a supplementary explanation of the reasons for the results about the order of attention layers of Table 3. \\\n**A2:** Thanks for the encouraging comments. Based on the reviewer’s comments, we added more description and analysis of Table 4. In Table 3, we analyze the effect of the different order of attention layers. Specifically, the best two orders are “N->V->T” and “N->T->V”. It demonstrates that neighbor information provides more personalized information for entity alignment which is consistent with existing works. When the neighbor multi-head attention is introduced in the last, the performance drops dramatically. It demonstrates that early introducing neighbor information can provide more directional information for effectively aggregating textual and visual information. In addition, the order of visual and textual information plays a relatively small role which is likely to be related to the dataset. We will give a supplementary explanation of the reasons for the results of Table 3 in Section 4.6.\n\n**Q3:** Add the Related Work to the main text. \\\n**A3:** Thanks for the kind suggestion and we will move the related work from the appendix to the main text.\n\n**Q4:** How is the problem addressed in this paper different from others? \\\n**A4:** Existing works mainly focus on aggregating multi-modal information while ignoring whether the information should be aggregated directly, which could lead to incorrect alignment. Different from the previous works, we argue that a differentiated and personalized aggregation of neighbors and modalities should be performed. It helps to prevent the misalignment of cross-modal semantics and the non-discriminative representations of different types of entities.\n\n**Q5:** Why does “N-V-T” work better? Is it related to the dataset? \\\n**A5:** We do experiment on another MMEA dataset FB15K-YAGO15K (80%) . The experiment shows that the best order is also“N-V-T”. The experiment also demonstrates that early introducing neighbor information can provide more directional information for the multi-modal entity alignment task. We will add it in Section 4.6.\n\n\n| **Variants** | **MRR** | **Hits@1** | **Hits@10** | **$\\Delta$ Avg**  |\n| :---: | :---: | :---: | :---: | :---: |\n| **Meaformer (N->V->T)**  | **76.9** | **68.9** | **88.4**  | - |\n| Meaformer (N->T->V) | 76.5 | 68.3 | 87.8 | 0.5 $\\downarrow$ |\n| Meaformer (V->N->T) | 76.1 |  67.7 | 87.5 | 1.0 $\\downarrow$ |\n| Meaformer (V->T->N) | 75.0 | 67.1 | 86.7 | 1.8 $\\downarrow$ |\n| Meaformer (T->N->V)  | 75.8 | 67.4 | 87.1 | 1.2 $\\downarrow$ |\n| Meaformer (T->V->N) | 75.3 | 66.8 | 86.4 | 1.9 $\\downarrow$ |\n\n\n**Q6:** Typos: Line 59, multi-mdoal. Line 176, Meaformer also design. Line 449, Comparision. \\\n**A6:** Thanks and we will fix the above typos and grammar problems.\n\n**Q7:** Consistency: multi-modal and multimodal, MMKG1 and MMKG 1. \\\n**A7:** We will change to multi-modal and MMKG 1 to make consistency.\n\n**Q8:** Improvements: Caption of Figure 4, maybe “… means interference WITH some proportion of attributes or neighbors.” \\\n**A8** Yes. We will fix the sentence to “… means interference with some proportion of attributes or neighbors.”.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "There are three reasons:\n\nA. Improved attention mechanism in KG transformer for MMEA. It is efficient and natural to capture heterogeneous information separately in stages/blocks. \n\nB. The idea of prefix injection is interesting. The type prefixes are injected into both self-attention and FNN, which is helpful for MMEA task. Beyond that, such an idea may inspire others to explore new injection forms of other prefixes in other tasks.\n\nC. SOTA results of sufficient experiments.\n", "reasons_to_reject": "There are also three reasons:\n\nA. The problem of “incorporating/aggregating the different types of information” should be stated more clearly, rather than just providing some examples. I believe this is the motivation for designing the new self-attention.\n\nB. As the major contribution of this paper, the order of attention layers is not fully analyzed in experiments. It is recommended to make a supplementary explanation of the reasons for the results of Table 3.\n\nC. In my opinion, the analysis of Related Work is important for this paper. It is recommended to add the corresponding section to the main text.\n", "questions_to_authors": "1. How is the problem addressed in this paper different from others?\n\n2. Why does “N-V-T” work better? Is it related to the dataset?", "paper_topic_and_main_contributions": "This paper focuses on multi-modal entity alignment (MMEA). To make better use of different information of MMKGs (neighbors, attributes, and types in this paper), the authors design a prefix-injected MMKG transformer with a hierarchical modifiable self-attention. The experimental results show the effectiveness of the proposed model.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Typos: Line 59, multi-mdoal. Line 176, Meaformer also design. Line 449, Comparision.\n\nConsistency: multi-modal and multimodal, MMKG1 and MMKG 1.\n\nImprovements: Caption of Figure 4, maybe “… means interference *WITH* some proportion of attributes or neighbors.”\n"}}, {"review_id": "pkAKh3IzRn", "reviewer": "Reviewer_adWR", "summary": "", "questions": "1. What does \"one-dimensional vector\" mean in 186? Is it a real? \n\n2. What does the colours of the entities in Figure 1 mean? They could be specified in the caption. ", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** The technical contribution of the proposed method, which uses a Transformer for multi-modal KG EA, is only incremental. \\\n**A1:** In this paper, we are not simply employing a vanilla Transformer. On the contrary, we carefully reform the three core components of the Transformer architecture specifically for the MMEA task, including the position encoding, self-attention mechanism, and feed-forward network. First, we introduce the modality and structural information into the position embedding which is critical for the MMEA task. Second, we propose a carefully-designed hierarchical modifiable self-attention mechanism specifically to address the aggregation issue of different information in the MMEA task. Third, we propose a type injection in FFN to enable better filtering out of unsuitable candidates for entity alignment. The above technical contribution lies in the core components (i.e., position encoding, self-attention, and FFN) of the vanilla Transformer architecture. Therefore, we are trying to design a new Transformer for the MMEA task to overcome the aggregation issue, rather than simply using a vanilla Transformer in the task. We thank the reviewer for the question and will refine the contribution part to be clearer.\n\n**Q2:** The quality of the paper should be improved. Some textual descriptions in this paper are not clear, making it hard to follow. \\\n**Q2.1:** E.g., in 043 - 044, for “due to the heterogeneity of MMKGs (e.g., different neighbors, multi-modal attributes, distinct types)”, what does “distinct types” mean? \\\n**A2.1:** The distinct types mean the entity types. For each entity, we introduce a distinct type. We will change it to distinct entity types for ease of understanding.\n\n**Q2.2:.** In 230, what does “first-order neighbors” mean? \\\n**A2.2:** The “first-order neighbors” mean that entities directed connect to the current entity in the multi-modal knowledge graph. We will provide an explanation of this in Section 3.1.2.\n\n**Q2.3:** What does “we randomly initialize a reference order” mean? Many texts in this paper are hard to follow. \\\n**A2.3:** For an entity of the MMKG, there are multiple first-order neighbors. For the Transformer architecture, we should input it in a specific order. However, the order is not existing in the MMKG. Thus, we randomly initialize the order. We will give a more detailed explanation.\n\n**Q3:** Only two cross MM KG EA benchmarks are tested. There are more cross lingual MM EA benchmarks that can be tested. \\\n**A3:** Thanks for the advice. We supply the experiments on three cross lingual MMEA datasets  $DBP15K_{ZH-EN}$, $DBP15K_{JA-EN}$ and  $DBP15K_{FR-EN}$ with the same experiment settings in the existing papers [1-3]. From the results, our model performs best on MRR and Hits@10. The experimental results demonstrate that our model preserves the unique semantics of different information by introducing a hierarchical modifiable self-attention block, which is suitable for the cross lingual MMEA task. We will add the results in Section 4.\n\nThe experiment on  $DBP15K_{ZH-EN}$:\n\n| **Models** | **MRR** | **Hits@1** | **Hits@10** |\n| --- | :---: | :---: | :---: |\n| EVA [1] | 95.1 | 92.9 | 98.6 |\n| MCLEA [2] | 94.6 | 92.6  | 98.3 |\n| MEAformer [3]  | 96.5 | 94.9  | 99.3 |\n| **Ours** | **96.8** | **95.2** | **99.7** |\n\n\nThe experiment on $DBP15K_{JA-EN}$:\n\n| **Models** | **MRR** | **Hits@1** | **Hits@10** |\n| --- | :---: | :---: | :---: |\n| EVA [1] | 97.6 | 96.4 | 99.7 |\n| MCLEA [2] | 97.3 | 96.1 | 99.4 |\n| MEAformer [3] | 98.6 | 97.8 |  99.9 |\n| **Ours** | **99.0** | **98.1** | **100.0** |\n\nThe experiment on $DBP15K_{FR-EN}$:\n\n| **Models** | **MRR** | **Hits@1** | **Hits@10** |\n| --- | :---: | :---: | :---: |\n| EVA [1] | 99.5 | 99.0 | 99.9 |\n| MCLEA [2] | 99.2 | 98.7 | 99.9 |\n| MEAformer [3] | 99.5 | **99.1** | **100.0** |\n| **Ours** | **99.6** | 98.8 | **100.0** |\n\n[1]Visual Pivoting for (Unsupervised) Entity Alignment, Fangyu Liu, Muhao Chen, Dan Roth, and Nigel Collier, AAAI 2021. \\\n[2]Multi-modal Contrastive Representation Learning for Entity Alignment, Zhenxi Lin, Ziheng Zhang, Meng Wang, Yinghui Shi, Xian Wu, and Yefeng Zheng, COLING 2022. \\\n[3]MEAformer: Multi-modal Entity Alignment Transformer for Meta Modality Hybrid. Zhuo Chen, Jiaoyan Chen, Wen Zhang, Lingbing Guo, Yin Fang, Yufeng Huang, Yuxia Geng, Jeff Z. Pan, Wenting Song, Huajun Chen, arXiv 2022.\n\n**Q4:** What does “one-dimensional vector” mean in 186? Is it a real? \\\n**A4:** Thanks for the comment and it is a writing mistake. We correct the expression as “convert the matrix into a vector similar to word embeddings”.\n\n**Q5:** What does the colours of the entities in Figure 1 mean? They could be specified in the caption. \\\n**A5:** The meanings of different colors can already be found in the caption of Figure 1 in the submitted manuscript. We will refine the caption to make it clearer. Specifically, the different colors mean different types of information in an MMKG. The blue ones are entities, the green ones are textual attributes, and the orange ones are visual attributes.\n\n**Q6:** Missing References: The paper misses another method also named “Meaformer”. \\\n**A6:** Thanks for your advice. We will add this paper [3] to the related work. It dynamically predicts the mutual correlation coefficients among modalities for entity-level modality fusion. Different from the above paper, our paper focuses on designing an MMKG transformer within only one transformer encoder to maintain the structure information by position embedding and designing a hierarchical modifiable self-attention block. \n\n[3]MEAformer: Multi-modal Entity Alignment Transformer for Meta Modality Hybrid. Zhuo Chen, Jiaoyan Chen, Wen Zhang, Lingbing Guo, Yin Fang, Yufeng Huang, Yuxia Geng, Jeff Z. Pan, Wenting Song, Huajun Chen, arXiv 2022.\n\n**Q7:** In 175-176, “Meaformer also design” –> “Meaformer also designs”. \\\n**A7:** We will change the sentence to “Meaformer also designs”.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A Transformer-based multi-modal knowledge graph entity alignment method is proposed, with promising results achieved. ", "reasons_to_reject": "1. The technical contribution of the proposed method, which uses a Transformer for multi-modal KG EA, is only incremental. \n\n2. The quality of the paper should be improved. Some textual descriptions in this paper are not clear, making it hard to follow. E.g., in 043 - 044, for \"due to the heterogeneity of MMKGs (e.g., different neighbors, multi-modal attributes, distinct types)\", what does \"distinct types\" mean? In 230, what does \"first-order neighbous\" mean? What does \"we randomly initialize a reference order\" mean? Many texts in this paper are hard to follow. \n\n3. Only two cross MM KG EA benchmarks are tested. There are more cross lingual MM EA benchmarks that can be tested. \n\n", "questions_to_authors": "1. What does \"one-dimensional vector\" mean in 186? Is it a real? \n\n2. What does the colours of the entities in Figure 1 mean? They could be specified in the caption. ", "paper_topic_and_main_contributions": "This paper presents a Transformer-based method Meaformer for multi-modal knowledge graph entity alignment. It uses BERT and positional encodings to get embeddings, and feed these embeddings into hierarchical modifiable self-attention block. \n\nThe evaluation uses two cross-KG EA datasets FB15K-DB15K and FB15K-YAGO15K. The reported performance is better than the adopted baselines. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. In 175-176, \"Meaformer also design\" --> \"Meaformer also designs\". \n\n"}, "extra_text": {"Missing_References": "The paper misses another method also named \"Meaformer\" (https://arxiv.org/abs/2212.14454) which was online in December 2022. This works also uses Transformer, and the authors should compare the proposed method to it. "}}, {"review_id": "5xqBz7mzuT", "reviewer": "Reviewer_Up9k", "summary": "", "questions": "No questions.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** Lack of citations and comparisons to recent similar papers in entity alignment. If this latest relevant paper as well as similar papers could be cited, then I think this paper should be accepted as a good paper. \\\n**A1:** Thank the reviewer for your high acceptation of our work. LODEME[1] iteratively performs logical deduction and multi-modal embedding and developed a structure-aware attention mechanism such that the entity embeddings can incorporate multiple images with different emphases. Different from the above paper, our paper focuses on designing a Hierarchical Modifiable Self-Attention that hierarchically introduces neighbor features, multi-modal attributes, and entity types to better capture complex interactions between modalities and contextual information. Furthermore, entity alignment works [2-4] designed for the cross-lingual scenario advised by Reviewer adWR. We will add the citations and comparisons to recent similar papers in entity alignment on the related work.\n\n[1] Vision, Deduction and Alignment: An Empirical Study on Multi-modal Knowledge Graph Alignment. Yangning Li, Jiaoyan Chen, Yinghui Li, Yuejia Xiang, Xi Chen, Haitao Zheng, arXiv 2023.\\\n[2]Visual Pivoting for (Unsupervised) Entity Alignment, Fangyu Liu, Muhao Chen, Dan Roth, and Nigel Collier, AAAI 2021.\\\n[3]Multi-modal Contrastive Representation Learning for Entity Alignment, Zhenxi Lin, Ziheng Zhang, Meng Wang, Yinghui Shi, Xian Wu, and Yefeng Zheng, COLING 2022.\\\n[4]MEAformer: Multi-modal Entity Alignment Transformer for Meta Modality Hybrid. Zhuo Chen, Jiaoyan Chen, Wen Zhang, Lingbing Guo, Yin Fang, Yufeng Huang, Yuxia Geng, Jeff Z. Pan, Wenting Song, Huajun Chen, arXiv 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The strengths of this paper are as follows:\n1. Novel MMEA Framework: The paper proposes a novel MMEA framework called \"Meaformer,\" which leverages the multi-modal KG Transformer to effectively integrate heterogeneous information from different types of data in multi-modal knowledge graphs (MMKGs). This framework addresses the challenges posed by the presence of diverse information, such as neighboring entities, multi-modal attributes, and entity types.\n2. Hierarchical Modifiable Self-Attention Block: To handle task-related intra-modal features, the paper introduces a hierarchical modifiable self-attention block in the transformer encoder. This design preserves the unique semantics of different information types, enabling better integration of multiple information sources. This attention mechanism likely helps in capturing complex relationships and patterns within the MMKGs.\n3. Entity-Type Prefix Injection: The paper devises two entity-type prefix injection methods to incorporate entity-type information using type prefixes. By doing so, it restricts global information sharing for entities not present in the MMKGs. This approach likely improves the alignment task's accuracy by focusing on relevant entity types, enhancing the entity representations.\n4. State-of-the-Art Performance: The proposed Meaformer framework demonstrates state-of-the-art performance on public multi-modal entity alignment datasets. This indicates that the approach is highly effective in identifying equivalent entity pairs across MMKGs and outperforms existing strong competitors in the field.\nOverall, the paper's strengths lie in its innovative framework design, the use of hierarchical modifiable self-attention, and the effective incorporation of entity-type information, leading to superior performance in multi-modal entity alignment tasks.\n", "reasons_to_reject": "Lack of citations and comparisons to recent similar papers in entity alignment. If this latest relevant paper as well as similar papers could be cited, then I think this paper should be accepted as a good paper.", "questions_to_authors": "No questions.", "paper_topic_and_main_contributions": "This paper hierarchically introduces neighbor, multimodal attribute, and entity types to enhance the alignment task. The Meaformer framework introduces a novel approach called Multi-Modal KG Transformer (MMKT) to effectively integrate heterogeneous information. It incorporates a hierarchical modifiable self-attention block to establish associations of task-related intra-modal features through layered introduction, while also using an entity-type prefix to enhance entity representations. The proposed framework achieves remarkable performance on public multi-modal entity alignment datasets, outperforming state-of-the-art methods.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Vision, Deduction and Alignment: An Empirical Study on Multi-modal Knowledge Graph Alignment.", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A", "Justification_for_Ethical_Concerns": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper focuses on multi-modal entity alignment (MMEA). To make better use of different information of MMKGs (neighbors, attributes, and types in this paper), the authors design a prefix-injected MMKG transformer with a hierarchical modifiable self-attention. The evaluation uses two cross-KG EA datasets FB15K-DB15K and FB15K-YAGO15K. The proposed framework achieves remarkable performance on these datasets, outperforming state-of-the-art methods.
Decision: Accept-Findings