Entity Alignment with Noisy Annotations from Large Language Models
Authors: Shengyuan Chen
Qinggang Zhang
Junnan Dong
Wen Hua
Qing Li
Xiao Huang
Venue: NeurIPS 2024 poster (2024)
Keywords: Entity alignment
Large language  models
probabilistic reasoning
active learning
Primary area: natural_language_processing
TL;DR: LLM4EA is a novel framework for entity alignment with noisy annotations from large language models, by effectively learn from noisy annotations and optimizing query budget utility..

Abstract: Entity alignment (EA) aims to merge two knowledge graphs (KGs) by identifying equivalent entity pairs. While existing methods heavily rely on human-generated labels, it is prohibitively expensive to incorporate cross-domain experts for annotation in real-world scenarios. The advent of Large Language Models (LLMs) presents new avenues for automating EA with annotations, inspired by their comprehensive capability to process semantic information. However, it is nontrivial to directly apply LLMs for EA since the annotation space in real-world KGs is large. LLMs could also generate noisy labels that may mislead the alignment. To this end, we propose a unified framework, LLM4EA, to effectively leverage LLMs for EA. Specifically, we design a novel active learning policy to significantly reduce the annotation space by prioritizing the most valuable entities based on the entire inter-KG and intra-KG structure. Moreover, we introduce an unsupervised label refiner to continuously enhance label accuracy through in-depth probabilistic reasoning. We iteratively optimize the policy based on the feedback from a base EA model. Extensive experiments demonstrate the advantages of LLM4EA on four benchmark datasets in terms of effectiveness, robustness, and efficiency.

Entity Alignment with Noisy Annotations from Large Language Models
Shengyuan Chen
Qinggang Zhang
Department of Computing The Hong Kong Polytechnic University Hung Hom, Hong Kong SAR shengyuan. chen&connect. polpanneetpllllulu.. polyu.hk
Department of Computing
The Hong Kong Polytechnic University
Hung Hom, Hong Kong SAR
qinggangg
Srnggangg.zhnggoconneec zhang@connect. polpnnnettpplly.....e..e.. yolyu.hk hk
Junnan Dong
Wen Hua
Department of Computing
Department of Computing
The Hong Kong Polytechnic University Hung Hom, Hong Kong SAR hanson. dong@connect. wwwpooonnnntttnnto...ooooooommme polyu. hk
The Hong Kong Polytechnic University Hung Hom, Hong Kong SAR wency hua@polyu. edu. hk
Qing Li
Xiao Huang
Department of Computing The Hong Kong Polytechnic University Hung Hom, Hong Kong SAR csqli0comp polyu. edu, hk
Department of Computing
The Hong Kong Polytechnic University Hung Hom, Hong Kong SAR xi xiaohuang@comp polyu. edu. hk
Abstract
Entity alignment (EA) aims to merge two knowledge graphs (KGs) by identifying equivalent entity pairs. While existing methods heavily rely numan-generaan labels, it is prohibitively expensive to incorporate cross-domain experts for anno- tation in real-world scenarios. The advent of Large Language Models (LLMs) presents new avenues for automating EA with annotations, inspired by their com- prehensive capability to process semantic information. However, it is nontrivial to directly apply LLMs for EA since the annotation space in world KGs is large. LLMs could also generate noisy labels that may mislead the alignment. To this end, we propose a unified framework, LLM4EA, to effectively leverage LLMs for EA. Specifically, we design novel active learning policy significantly reduce the annotation space by prioritizing the most valuable entities based on the entire inter-KG and intra-KG structure. Moreover, we introduce an unsupervised label refiner to continuously enhance label accuracy through in-depth probabilistic reasoning. We iteratively optimize the policy based the feedback from base EA model. Extensive experiments demonstrate the advantages of LLM4EA on four benchmark datasets terms of effectiveness, robustness, and efficiency.
Introduction
Knowledge graphs (KGs) serve foundational structure for storing and organizing structured knowledge about entities and their relationships, which facilitates effective and efficient search capabilities across various applications. They have been widely applied in question-answering systems (Dong et al., 2023, 2024c), recommendation systems (Catherine & Cohen, 2016; Chen al., 2024a), social network analysis (Tang al., 2008), Natural Language Processing (Weikum & Theobald, 2010), etc. Despite their extensive utility, real-world KGs often suffer from issues
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

such as incompleteness. domain specificity, or language constraints, which limit their effectiveness in discippiipll or multilingual contexts. To address these challenges, entity alignment (EA) aims to merge disparate KGs into unified, comprehensive knowledge base by identifying and linking equivalent entities across different KGs. For instance, by aligning entities between a financial KG and a legal KG, EA facilitates the understanding of complex relationships, such as identifying the same corporations across the two KGs to assess how legal regulations impact their financial performance. This alignment enables a more nuanced exploration and interrogation of interconnected data, providing richer insights into how entities operate across multiple domains.
Entity aligmment models predict the equivalence of two entities by measuring their alignment probability. Specifically, rule-based methods (Suchanek et al., 2012; Jiménez-Ruiz & Cuenca Grau, 2011; Qi et al., 2021) utilize predefined rules or heuristics to update alignment probabilities and propagate alignment labels. Conversely, embedding-basee models seek to exploit advanced techniques in graph learning (Li et al., 2024; Liu et al., 2024b,a, 2023), parameterizing these probabilities using similarity scores between entity representations learned through knowledge graph embedding algorithms such as translation models (Chen et al., 2017: Sun et al., 2018) or Graph Convolutional Networks (GCNs) (Wu et al., 2019; Mao et al., 2021; Wang et al., 2018; Huang et al., 2023). However, these methods heavily rely on extensive and accurate seed alignments for training- a requirement that poses significant challenges. The need for substantial, cross-domain knowledge to annotate such alignments often makes their acquisition prohibitively expensive.
Recently, Large Language Models (LLMs) have showcased their superior capability in processing semantic information Dong et al. (2024a), which has significantly advanced various graph learning tasks such as node classification (Chen al., 2024d), graph reasoning (Zhao et al., 2023a; Chai et al., 2023), recommender systems (Zhou et al., 2022; Wu et al., 2023). SQL query generation (Zhang et al., 2024a), and knowledge graph-based question answering (Wang et al., 2024; Zhang et al., 2024b; Dong et al., 2024b). Their capacity to extract meaningful insights from graph data opens up new possibilities for automating EA. Notably, recent studies (Zhong et al., 2022; Zhao et al., 2023b; Jiang et al., 2024) have explored the use of LLMs in EA, primarily focusing on finetuning a pretrained LLM such as Bert to learn semantic-aware representations, relying on accurate seed alignments as training labels. Yet, the potential of LLMs for label-free EA via in-context learning remains unexplored.
However, directly applying LLMs to automate EA poses significant challenges. Firstly, conventional EA models presume that all annotations are correct; yet, LLMs can generate false labels due to LLMS' inherent randomness and the potential incompleteness or ambiguity in the semantic information of entities. Training an EA model directly on these noisy labels can severely impair the final alignment performance. Secondly, given the vast number of entity pairs, annotation with LLMs would be prohibitively expensive. Maximizing the utility of limited LLM query budget is essential. Existing solutions such as active learning cannot be directly applied since the annotations are noisy.
In response to the outlined challenges. we introduce LLM4EA, a unified framework designed to effectively learn from noisy pseudo-labels generated by LLMs while dynamically optimizing the utility of constrained query budget. LLM4EA actively selects source entities based on feedback from a base EA model, focusing on those that significantly reduce uncertainty for both the entities themselves and their neighbors. This approach allocates the query budget to important entities, guided by the intra-KG and inter-KG structure. To manage the noisy pseudo-labels effectively, LLM4EA incorporates an unsupervised label refiner that enhances label accuracy by selecting subset of confident pseudo-labels through probabilistic reasoning. These refined labels are then utilized to train the base EA model for entity alignment. The confident alignment results inferred by the EA model inform active selection in subsequent iterations, thereby progressively improving the framework's 's effectiveness in a coherent and integrated manner. Contributions are summarized as follows:
We propose LLM4EA, an in-context learning framework that uses LLM to annotate entity pairs. Leveraging the LLMs' zero- shot learning capability, this framework generates pseudo-labels, providing a foundation for entity alignment without ground truth labels.
Novel LLM-based framework for entity alignment:
Unsupervised label refinement: Our framework introduces an unsupervised label refiner informed by probabilistic reasoning. This component significantly improves the accuracy of LLM-derived pseudo-labels, enabling effective training of entity alignment models.
Active sampling module: We propose an active selection algorithm that dynamically searches entities in the huge annotation space. Guided by feedback from the EA model,

---

this algorithm adjusts its policy based on inter-KG and intra-KG structures, optimizing the utility of LLM queries and ensuring efficient use of resources.
Empirical validation and superior performance: We rigorously evaluate our framework through extensive experiments and ablation studies. The results demonstrate that L.L4EAA not only outperforms basclines by large margin but also shows robustness and efficiency. Each component of the framework is shown to contribute mcaningfully to the overall performance, with clear synergistic effects observed among the components.
2 Problem definition
knowledge graph 9 comprises a set of entities &, a set relations R. and a set of relation triples T where each triple (en,r, ex) T represents directional relationship between its head entity and tail entity. Given two KGs 9 = {E, R. 9 = (E), , R', , T'} and fixed query budget B to Large Language Model, aim to train infer the matching score m e') for all entity pairs {(e, e'), E E, utilizes ground truth alignment set A assess the prediction accuracy for target entities in both directions, (e, ?) and (?, e') for each true pair (e, E A. based on the ranked matching scores mg. Evaluation metrics are hit@k (where (1, 103) and mean reciprocal rank (MRR).
entity alignment model 0 based the LLM''s annotations to
''' E }. The evaluation process
3 Entity alignment with noisy annotations from LLMs
We aim to design a framework to perform entity alignment with LLMs. Our design is motivated by the following insights. Firstly, we have huge search space (the overall annotation space is ((E))(")) to identify the core entity pairs to annotate. Secondly, we don't 't know whether the annotated labels correct not, because have prior knowledge hcuristic of the label distribution. Finally, we perform annotations iteratively, requiring the model to adjust its search policy based on annotation effectiveness, while we have verifiable feedback of this annotation accuracy.
Based on these insights, we propose LLM4EA- an iterative framework that consists four intercon- nected steps in each cycle, as illustrated in Figure 1. Initially, an active selection technique optimizes the use of resources by choosing critical source entities that significantly reduce uncertainty for themselves and their neighbors. Subsequently, an LLM- annotator identifies the counterparts for the selected source entities, generating a set pseudo-labels. Next, label refiner improves label accuracy by eliminating structurally incompatible labels. This process involves formulating a combinatorial optimization problem and utilizing a bubbublistic-rrasonin based greedy search algorithm to efficiently find local-optimal solution. Finally, these refined labels are used to train base EA model for the entity alignment task. The outcomes of the alignment then serve as feedback to inform subsequent rounds of the active selection policy. Further details are provided below.
Active sampling
Probabilistic reasoning for label refinement
Active selection
Generole primpts DOOD
mede
Source KG
FA
LLM
Sapa Upbbat
Selected
Pscude-labels
Refined labels
Target
Aliigment probabilities
Figure 1: Overview of the LLM4EA framework. LLM4EA utilizes active sampling to select important entities based on feedback from an EA model. It also includes a label refiner to effectively train the base EA model using noisy pseudo-labels. Feedback from the EA model updates the selection policy.
3.1 Active selection of source entity
We aim to maximize the utility of the budget by actively allocating the budget to those beneficial entities. To do this, we sample source entities that reduce the most uncertainty of both themselves and their neighboring entities, by dynamically adjusted policy. The measurement of uncertainty
3

---

reduction is based on two assumptions: 1) an entity's own uncertainty is inversely proportional to its alignment probability with its most probable counterpart; 2) the amount of uncertainty an entity eliminates for its neighbors is closely linked to the relational ties between them. To systematically assess this, we introduce the concept of relational uncertainty, quantified as follows:
(e,) (1 P(en)) +
Σ
w, (1 (1- P(e)).
(I)
Here, Wp is weight coefficient reflecting the significance of relation r and signifies how much en contributes to reducing the uncertainty of e, through the relation r. For this purpose, we employ functionality F(r) (formally defined in Eq. (4)) as the weight wy, as it quantifies the uniqueness of the tail entity for a given specified head entity. P(e)= == maxe'eg' P(e e') represents the alignment probability of the top-match entity for e. These alignment probabilities P(e = e') are obtained through probabilistic reasoning during label refinement (Section 3.3.2) and are augmented by the inferred alignments from the base EA model (Section 3.4). In the initial iteration, all alignment probabilities are set to O.
It's important to note that some source entities are linked to large number of uncertain neighbors (those with low P(e)). These source entities are crucial but may be overlooked if their connected relations have low functionality. Hence, we introduce neighbor uncertainty as another metric to assess an entity's importance, by removing the functionality-baat bbased weight coefficient:
(e())) = (1 P(en)) + Σ (1-P(e)))
(2)
(EL.REET
To integrate these two metrics, we employ rank aggregation by mean reciprocal rank:
U(en)= = x (11 + 1
(3)
(en) (un(en))
Here, Tur (cx) and Fan (cn) denote the ranking of e₁ when using U, and U, as metric, respectively. This simple-effective aggregation technique is advantageous for our task since it's scale invariant and requires no validation set for tuning hyperparameters, making it more practical in this task.
3.2 LLM as annotator
Counterpart filtering. With the selected source entities, we employ an LLM annota- tor to identify the counterpart from E' for each source entity, generating a set of pseudo-labels L = f e')le E, E'). To narrow down the search space, first filter out the less likely counterparts before querying the LLM, selecting only the top-k most similar counterparts from 8 The similarity metric is flexible: we use a string matching score based on word edit distance, but other methods are also viable, such as semantic embedding distances derived from word embedding models. By adjusting k we can trade-off between the recall rate of counterparts and the query cost.
Prompt design. There are primarily two methods for retrieving context information to construct textual prompts: randomly generated prompts and dynamically tuned prompts. The former involves randomly selecting neighbors to construct contexts for the entity, while the latter dynamically selects neighbors based feedback from the EA model. For fair comparison, prompts across all baselines and the proposed LLM4EA. These prompts include the name of each entity and set of relation triples three randomly selected neighbors. For the baseline models, pseudo-labels are generated at once and used for training. For LLM4EA, we evenly divide the budget B into iterations and generate pseudo-labels each iteration using the allocated B/n budget.
randomly generated
3.3 Probabilistic reasoning for label refinement
The pseudo-labels generated by the LLM can be noisy, and directly using these labels to train an entity alignment (EA) model could undermine the final performance. Although estimating the label distribution by asking the LLM for confidence scores or querying multiple times to measure consistency are potential solutions, these approaches can be vulnerable or introduce additional costs.
In light of this, we propose a label refiner that leverages the structure of knowledge graphs. The refinement process is framed as combinatorial optimization problem aimed at minimizing overall

---

structural incompatibility among labels. Utilizing probabilistic reasoning technique, we progres- sively update our confidence estimation for each label and select those that are mutually compatible, ultimately producing a set of accurate pseudo-labels. Detailed explanations follow below.
3.3.1 Functionality and probabilistic reasoning
Functionality. The functionality of a relation quantifies the uniqueness of tail entities for specified head entity, calculated the ratio of unique head entities total head-tail pairs linked by the relation. Conversely, inverse functionality quantifies the tail entity uniqueness for a specified head entity. Formally, these are defined as:
F(r) = Heanlee, r, E T3 (e)(e,, E TTH
(r) Hes,(en, E T3 H[en, ex) (eh, ex) E TT)
(4)
For instance, suppose a KG contains two triples for the relation locate_in: in: (Hawaii, locate_in, US) and (M locote_ iin, US). Then F(locate_in) words, given ( (M Miami, locate_iie ?), the answer for the missing tail entity is unique; while given (?, locate in, US, there are multiple answers for the missing head entity. Such relational patterns are useful for identifying an entity based on its connections within the intra-graph structure.
1.0 and (=ooaacte__nn
0.5. In other
Probabilistic reasoning. If two entities are each connected to entities that are aligned across KGs, this increases the likelihood that they should be aligned as well. Based on this heuristic, an entity pair's alignment probability P(en c/)) can be inferred by aggregating its neighbors' alignment probability via relation functionality:
1
II (1-F-1())PC (P))pP))(EE =c)) x -1- (1-F-(())P(+ Pr) = c)). (5)
(e).. (e))E
Here, P(r S r') denotes the probability of being subrelation of r', estimated by alignment probabilities of connected entities:
Σ -e MeDeT (1- P(e, cn)
Σ (1- (1 = ee) = en)
These formulations allow for the propagation and updating of aligmment probabilities in a manner that is cognizant of relational structures. We employ this technique to design label refiner below.
3.3.2 Label refiner
Label incompatibility. We exploit the "incompatibility" of labels for label refinement, based on the assumption that correct labels can infer each other, while false label could be incompatible with its correctly aligned neighbors. We define the overall incompatibility on label set L as:
(C) = Σ
(Ir(e)=e)) smax P(a,e)) + 1 =e smaax Plowse))
(Es.ef)EL
Here, 1 P(ex = P(e.e)) 1 if en is not the top-match for e/, otherwise O. It's important to note that detected incompatibility doesn't 't necessarily indicate the false alignment of (ex, e¹,)): it may suggest a misalignment of their neighbors. Given this, the key to label refinement is to jointly optimize the label's overall incompatibility while avoiding accidentally filtering out correct labels.
Objective. To enhance label quality, we propose to refine the laeudo-label set by finding a subset L C. L that minimizes its overall incompatibility; L- = argmingccd 4 (CC). Noteworthy that trivial solution for this optimization problem only preserving a set of isolated labels, such that max.gg P(e = e(,) 0 and max P(e, = e') = O for all (Cx, e()) E L' This trivial solution would lead to the exclusion of most accurate labels, an outcome we aim avoid. Considering this, we introduce an /1 penalty term to penalize the removal of labels, leading to our overall objective:
J argming'cc (D(C') + XlC C(').
(8)
5

---

Here À > 0 is a weight coefficient. Solving the above combinatorial problem is intractable as it requires computing ((')) for each possible set L' C C. which is NP-hard. Below we propose to search for a local-optimal solution by a greedy algorithm powered by probabilistic reasoning.
Greedy search. The algorithm begins by initializing the alignment probability P(e = e') = 6o for every pair (e, e") within the set L, where dg is a constant within the range (0,1). It then iteratively performs a search for an optimal label set L' through a series of voting steps. Each iteration is comprised of two main steps: probabilistic reasoning and label adjustment.
During the probabilistic reasoning step, the alignment probabilities and subrelation probabilities updated according to Eq. (5) and Eq. (6), respectively. This update process refines our estimates of label confidence based on the latest information. During the label adjustment step, the label set L' is updated based on these updated probabilities. Labels are appended to L' if their updated alignment probabilities exceed 5p, indicative of high confidence in their alignment, supported by their neighbors:
t L' ((e,, </) E (|e e > 60}
Conversely, labels demonstrating structural incompatibilities are excluded from L':
L' -' (en (en, (en, E P(e <l,) < max
mmxx P(e, bh); max P(ez, e) ) 3EE'
(10)
e'E8'
In this manner, labels are removed if they are incompatible with updated aligned neighbors, ensuring the preservation of only the most confident pairs within L'. To further refine the search process in subsequent iterations, we augment all entity alignment probabilities within L' to a superior score:
P(e = e') + max (P(e = 6)), for each (e, E L'.
(11)
Here 61 (do, 1) serves new threshold, elevating the alignment probabilities of confident pairs to foster a more directed and effective search. After ni, iterations, we get a set of confidently selected labels L that have high compatibility. The detailed algorithm is presented in Appendix A.2, and analyses of parameter efficiency and computational efficiency are provided in Appendix A.3.
3.4
Entity alignment
With the refined labels, train embedding based EA model to learn structure- aware represen- tations for each entity. After training, the EA model computes a matching score my (e, e') for each entity pair e') for evaluation. The selection of the base EA model is flexible, tailored to the task requirements. We chose a recently proposed GCN-based model, Dual-AMM (Mao et al., 2021), for its effectiveness and efficiency.
Feedback from the base EA model is crucial for dynamic update of the active selection policy. To generate effective feedback, infer high-confidence pairs (e, e') with the trained EA model, by selecting the pairs that both entities rank top for each other. These pairs are injected into the probabilistic reasoning system. Similar to the label refinement process, this system initializes with an alignment probability of 6 for these pairs and updates the estimation of alignment and subrelation probabilities using Eq. (5) and Eq. (6). The updated probabilities used to construet the uncertainty terms (i.e., U, and U,) to inform the active selection policy in subsequent iterations, thereby optimizing the budget utility and improving final performance continuously.
4 Experiments
In this section, we conduct experiments to evaluate the effectiveness of our framework. We begin by introducing the experimental settings. Then, we present experiments to answer the following research questions: RQ1. How effective the overall framework? RQ2. What is the impact of the choice of LLM on the cost and performance of LLM4EA? RQ3. What is the effect of the label refiner? RQ4. What is the impact of active selection?
4.1 Experimental setting
Datasets and LLM. In this study, we use the widely-adopted OpenEA dataset (Sun al., 2020), including two monolingual datasets (D- (D-W-15K and D-Y-15K) and two cross-lingual datasets (EN- DE-15K and EN-FR-ISE 15K). OpenEA comes in two versions: "V1" the normal version, and "V2"

---

the dense version. We employ "V2" in the experiments in the main text. The LLM version in this experiment is GPT-3.5 s.pt-3.-tturbo-0111 and GPT-4 sept-4-turbo-022--000 overall query budget is B = 0.
By default, the
Baselines. Baseline models include three GCN-based models GCNAlign (Wang al., 2018), RDGCN (Wu et al., 2019), Dual-AMN (Mao et al., 2021), and three translation- baseed models IMUSE (He al., 2019), AlignE, BootEA (Sun et al., 2018), Here, BootEA a variant of AlignE that adopts a bootstrapping strategy, equipped with label calibration component for improving the accuracy of bootstrapped labels. Baseline models are directly trained on the pseudo-labels generated by the LLM annotator, without label refinement or active selection. Every experiment is repeated three times to report statistics.
Setup of LLM4EA. We employ GPT-3.5 as the default LLM due to its cost efficiency. Other parameters = Rir, ks = 20, 6g 0.5, 61 0.9.
4.2 Results
4.2.1
Comprehensive evaluation of entity alignment performance
Table 1: Evaluation of entity alignment performance, measured by Hit@K for K E (1, 10), and Mean Reciprocal Rank (MRR), presented in %. Experiment statisties are computed over three trials.
EN-FR-15K
EN-DE-15K H
D-W-15K Hire 10
D-Y-15K Him@
Hire IO
MRR
Hir@
MRR
Hie@
MRR
Hir@
MRR
Groupl. Entity Alignment with GPT-335.
IMUSE
50.0a0.1
72.640.8
57. 5..5
51.6a4.7
75.9239
60.5a4.5
6.0a0.2
14.6:2.5
9.0+1.0
54.4a2.5
78.9&1.1
63.2+2.0
AlignE
6.620.3
24.5a0.5
12.620.5
6.200.3
18:421.0
10.420.5
8.010.9
24.0+2.7
13.3a1.4
50.1=2.0
76.621.4
59.2:1.8
BootEA
4488:1.1
71.9+1.2
54.2+1.2
68.1a0.2
85.420.3
74.340.2
60,840.2
79.340.1
67.420.2
87.840.1
96.740.1
91.2.0.1
GCNAlign
17.420.3
43.2a0.4
25.920.3
22.240.2
46.2+1.1
30.3+0.3
16.910.1
39.340.3
24.320.1
45.320.4
68.310.6
53.320.5
RDGCN
69.3=0.3
82,540.3
74.3+0.3
73.3+4.3
84.6+26
4734
79.2+0.7
89.740.5
83.240.6
82.643.7
91.9=1.3
86.1
Dual-AMN
51.9=0.3
79.640.9
61.6+0.5
70.5+0.7
91. 1+0.3 78.9+0.6
62.040.1
86.8+0.1
71.9+0.1
85.8+0.3
98.4+0.0
4+001
LLM4EA
74.260.3
92.9+0.4
81.0=0.3
89.1+0.5
97.8+0.1
92.640.3
87.5+0.3
96.7+0.1
90.9:0.2
97.7+0.0
99,5+0.0
98.3+0.0
Group2. Entity Alignment with GPT-4.
IMUSE
52.7=0.9
7..9+1.0
59.8+0.9
59.6+2.6
81.8+1.5
67.9+2.1
21.646.1
50.0=10.0
31.1+7.4
86.6+0.5
94.2+0.1
89.2+0044
AlignE
46.45..2
76.5&3.8
56.6a4.8
67.8:3.6
46.7a3.7
86.4+0.9
97.0s0.3
BoodFA
58.2=0.3
83.740.3
67.0+0.3
80.5+0.4
92.6+0.2
84.8+0.3
71.640.2
88.340.2
77.6+0.2
95.0+0.1
98.6+0.0
96.3+0.1
GCNAlign
41.9:0.4
6.0.5
51.240.4
61.640.1
41.4s0.2
82.640.2
94.920.2
87.2+0.1
RDGCN
72.120.2
84.520.1
76.720.2
74.121.1
85.120.7
78.021.0
82.5+1.1
91.420.7
85.921.0
65.420.9
93.2+0.4
88, 310.8
Dual-AMN
76.7=0.1
94.9H0.3
83.620.2
90.7=0.1
97.960.2
6.0.1
81,540.1
94.9+0.2
86.7=0.1
97.5+0.0
3+0.0
18.0
LLM4RA
80.2+0.3 96.0H0.2
86.0=0.2
93.1
98.7+0.2 95.340.3
89.8+0.3
97.9+0.2
92.9.0.3
97.940.1
99.6+0.0
98.540.1
To answer RQ1 and RQ2, we conducted two groups of experiments on OpenEA datasets, using GPT- 3.5 and GPT-4 the annotator, respectively. Results presented in Table 1. We also investigated the performance-cost comparison between the GPT-3.5 annotator and the GPT-4 annotator, illustrated in Figure 2. To control the randomness introduced by the LLMs, each experiment times to report mean and standard deviation. These results lead to several key observations:
repeated three
First, LLM4EA surpasses all baseline EA models, which are directly trained on the pseudo- labels, by a large margin. This can be attributed to 1) our label refiner's capability in filtering out false labels, reducing noise during training and enabling more accurate optimization towards the ground true objective; 2) our active selection component's 's ability to smartly identify important entities to annotate, which takes full advantage of the fixed query budget.
Second, using the GPT-4 results in higher performance than using the GPT-3.5 as the annotator. This observation conforms to the fact that GPT-4 is more advanced LLM with higher reasoning capacity and stronger semantic analysis, resulting in more precise annotation results and higher recall, thus providing more labels of high quality. We also observe that translation- based models (e.g.. AlignE) are sensitive noisy labels under GPT-3.5, while state-of-the -orte--rrrr GCN-based models (e.g.. RDGCN and Dual-AMM) are more robust. BootEA also demonstrates superior performance and robustness, attributed to its bootstrapping technique and enhanced by its capability in calibrating bootstrapped labels. However, its label calibration is only applied to the bootstrapped labels, so it still suffers from the false training labels. Our proposed LLM4EA, on the other hand, refines the label accuracy before training the EA model, thus ensuring more accurate training.
7

---

OFT-22
GFT322
G7T32
G7731
U714
UP14
G7T4
G774
Quary
Figure 2:
Performance-cost comparison between GPT-3.5 and GPT-4 as the annotator, evaluated by MRR. We increase the budget for GPT-3.5 to evaluate its performance. [nx] denotes using n. the default query budget. Each experiment is repeated three times to show mean and standard deviation.
Finally, LLM4EA is noise adaptive, enabling cost-efficient entity alignment. To further investigate the effect of the choice of L.L.,, we examined the performance-cost comparison between GPT-3.5 and GPT-4 as the annotator. We illustrate MRR in Figure 2 (detailed results are available in Appendix B.3). The results show that, by increasing the query budgets (measured by the number of tokens) for GPT- 3.5, the performance gradually increases. When the budget is 2x that of GPT-4, the performance is comparable or exceeds the performance of using GPT-4 as the annotator. According the pricing scheme of OpenAI, the input/output cost for million tokens for GPT-3.5 and GPT-4 is $0.50/S1, and $10/S30, respectively. This means that our noise- coise-adaptive framework enables cost-efficient entity aligmment with less advanced LLMs at 10x less actual cost than using more advanced LLMs, simply by increasing the token budget for the less advanced LLMs.
4.2.2 Effeec of the label refiner
Refined
64-
0.
0.2
EX-FR-15X
EN-ER-155 15K
Fined Bodast Raad 17
60
L1N D0-25k
0.0
IN-01-11
0.5
12
15
12
15
0.1
63
0.
Ierations
Iterations
Initial TPR
Figure 3: Analysis of the Label Refinement. We illustrate the evolution of the true positive rate (TPR) (left) and recall (middle) for refined labels across four datasets. Furthermore, we assess the robustness of the label refinement process by examining the TPR of refined labels against varying initial TPRs within the D-W-15K dataset (right), with initial pseudo-labels synthesized at different TPR levels.
To answer RQ3, we first analyze the evolution of the True Positive Rate (TPR) and the recall rate of the refined labels. Specifically, at each label refinement iteration, the TPR calculated as and the recall is calculated as LAnE') TArT2 The left and middle subfigures of Figure demonstrate how our label refiner progressively discovers accurate labels and optimizes the TPR. Initially, the TPR of the refined label set high (approximately 1.0), then decreases by certain percentage, and eventually increases again to a high TPR. We attribute this pattern to: 1) the most confident labels being discovered in the earliest itcrations, which are obvious alignments with many connected alignments; 2) as the algorithm progresses, some false pseudo-labels being erroneously added to the label set C': 3) as the label refinement continues, L' is adjusted and the false pseudo-labels are replaced with the correct labels inferred by the updated probability in Eq. (10).
|Anc'|
[27
Furthermore, we assess the robustness of our label refiner, depicted the right subfigure of Figure 3. We synthesize noisy labels and evaluate the output TPR in relation to varying input TPR levels, using two experimental schemes: fixed budget, where the budget remains constant 0.1 while the TPR changes, and fixed TP, where the number of true positives is fixed but the TPR and corresponding budgets are adjusted. The results demonstrate that the label refiner consistently elevates the TPR to over 0.9, even when the initial TPR is around 0.5, showcasing its high

---

robustness to noisy pseudo-labels. This result also reveals why our framework demonstrates robust performance with the less advanced GPT-3.5 annotator.
4.2.3 Ablation study
Table 2:
Ablation study overview. The table presents the performance of the LLM4EA (Ours) with various modifications. Group 1: removing the label refiner (w/o LR) and the active selection component (w/o Act); Group 2: replacing the active selection technique with relational uncertainty (-ru), neighbor uncertainty (-nu), degree (-degree), and functionality sum (-funcSum).
EN-FR-I5K
EN-DE-ISK
D-W-I5K
D-Y-I5K
Hir01
Hice 10
MRR
Hit@1
Hite 10
MRR
Haue 10
MRR
Hire1
Hire 10
MRR
Ours
74.240.3
92.920.4
81.040.3
89.120.5
97.820.1
92.620.3
87,5+0.3
96.740.1
90.90022
97.720.9
99,520.0
98.320.0
w/o LR
51.6
80.2:0.7
61.9408
74.421.7
94.200.6
32.621.
39.2+1.4
75.7+0.7
52.91.2
85.3+1.0
99.2+0.1
91.5+0.5
wlo Act
68.1+2.1
88.4+1.7
75.422.0
78.4+08
93.9+0.3
84.6=0.6
82.8+0.7
92.5+0.6
86.3+0.6
97.540.1
99.240.3
98.1+0.1
Ours-ru
70.811.0
91.220.4
78.240.8
83.920.3
97.7 89.620.2
88.720.6
97.420.3
92.1+0.5
97.740.0
99,410.1
98.3+0.0
Ours-nu
74.5a0.7
93.1=0.5
81.2+0.6
8=0.2
96.700.3
91.8-033
85. 1+0.5
95 2+0.5
88
9+0.5
97,6+0.1
99. 09,4+0.0
98, 2+0.0
Ours-degree
73.6+2.6
92.5+0.8
80.4+2.0
88.4=0.1
96.6+0.2 91.5=002
1+3.7
90.9+2.2 84.0+3.2
97.240.2
99.040.1
97.940.1
Ours: FuncSum
59.5a0.6
78.820.5 66.340.6
81.2+0.5
96.020.3 87.1=0.4
83.920.9 93.1 87.3+10
97.540.1
99.440.1
98, 140.1
Ablation studies detailed in Table answer RQ4 and reveal several key insights: Necessity of the Label Refiner for Effective Active Selection: The performance of "w/o LR", which lacks label refiner, is inferior not only to other model variants but also to the base model, Dual-AMMN. This underscores that active selection depends crucially on reliable feedback, which is compromised when the label refiner is absent; 2) Contribution of Relation and Neighbor Uncertainty Active Selection: Incorporating both relation and neighbor uncertainties significantly enhances the utility of the budget. Methods like "Ours-degree" and "Ours-funcSum" foeus only on their connections to neighbors and ignore the uncertainty of neighbors. In contrast, "Ours-ru" and "Ours-nu", which take these uncertainties into account, exhibit superior performance. This underscores the importance of considering neighbor uncertainty for effective active selection; 3) Robust Active Selection through Combined Metries: Our active selection approach integrates both relation uncertainty and neighbor uncertainty to enable robust active selection. By employing rank aggregation, it prioritizes entities that are deemed significant by both metrics, ensuring a more effective and nuanced selection process.
4.2.4
Pareto frontier of runtime overhead against performance.
EN-FR-15K
EN-DE-I5K
D-W-15K
D-Y-I5K
82-
80-
93
99-
MIGE
900
MBG
94
72
91
70-
10
Iteetoons
Deenttionss
Ite
Figure 4: Performance of entity alignment across four datasets with varying active sampling iterations, under a fixed query budget.
The runtime overhead is directly proportional to the number of active selection iterations, since each iteration involves a subsequent label refinement process. To explore the runtime-pert perforanpe trade-off in the LLM4EA approach, we examine the Pareto frontier of runtime versus performance. We conduct entity alignment experiments with a fixed query budget, varying the number of active selection iterations. The results of these experiments are illustrated in Figure 4.
The results indicate that performance initially increases as the number of iterations rises from to around 3, but further increases beyond this point lead to decline. This pattern can be attributed to two main factors: (1) More iterations allow for extensive learning from feedback during the active selection phase. (2) However, when iterations are excessively high, the number of generated labuds per iteration becomes small, leading to isolated pseudo-labels that undermine the label refinement process. These findings suggest that an optimal balance between runtime efficiency and performance can be achieved without excessive trade-offs, indicating a specific threshold for iterations beyond which no further performance gains observed.

---

5
Related work
LLMs for Entity Alignment. Recent approaches have sought to leverage LLMs for entity alignment in knowledge graphs, primarily focusing on integrating structural and semantic information for improved alignment performance. BERT-INT (Tang et al., 2020) fine-tunes a pretrained BERT model to capture interactions and attribute information between entities. Similarly, SDEA (Zhong et al., 2022) employs a pretrained BERT to encode attribute data, while integrating neighbor information via a GRU to enhance structural representation. TEA (Zhao et al., 2023b) reconceptualizes entity alignment as a bidirectional textual entailment task, utilizing pretrained language models to estimate entailment probabilities between unified textual sequences representing entities. A novel approach, ChatEA (Jiang ct al., 2024), introduces a KG-code translation module that converts knowledge graph structures into a format comprehensible to LLMs, enabling these models to apply their extensive background knowledge to boost the accuracy of entity alignment. Notably, these models primarily focus on fine-tuning pretrained language models using a set of training labels and do not exploit the zero-shot capabilities of LLMs. In contrast, our proposed model, LLM4EA, leverages the zero-shot potential of LLMs, enabling it to generalize to new datasets without the need for labeled data.
Probabilistic Reasoning. In the literature on knowledge graph reasoning, probabilistic reasoning has been effectively applied to infer new soft labels and their associated probabilities from existing labels. has been utilized in domains such as knowledge graph completion (Qu & Tang, 2019; Zhang et al., 2020; Fang et al., 2023; Chen et al., 2024b) and entity alignment (Suchanck et al., 2012; Qi et al., 2021; Liu et al., 2022: Chen et al., 2024c), where it naturally represents complex relational patterns with simple rules and performs precise inferences. In this work, however, due to the potential inaccuracies in the pseudo-labels generated by LLMs, the newly inferred alignments may be incorrect. Consequently, we opt not to employ probabilistic reasoning directly for the entity alignment task. Instead, we emphasize its use primarily for filtering false pseudo-labels labels that demonstrate structural incompatibilities within our framework. For completeness, include the results of comparison with a probabilistic reasoning model PARTS (Suchanek et al., 2012) in Appendix B.4.
6 Limitations
Firstly, during active selection, we distribute the query budget evenly for each selection rather than dynamically customizing the budget allocation for each iteration. This allocation approach could be improved by developing a more adaptive budget allocation strategy. Secondly, the framework currently does not provide direct support for temporal KGs. Although the probabilistic reasoning and active selection components inherently support entity alignment on evolving KGs, the base EA model is transductive. This necessitates retraining the model whenever new entities and relation triples are introduced into the KGs. However, this can be complemented by the research line of inductive entity alignment, such path-based embedding models or logic-based models, which can generalize to previously unseen entities without the need for retraining.
Conclusions
In this paper, we address the challenge of automating entity alignment with Large Language Models (LLMs) under budget constraints and noisy annotations. We introduce LLM4EA, a framework that maximizes the utility of fixed query budget using active sampling and mitigates erroneous labels with a label refiner employing probabilistic reasoning. Empirical results show that LLM4EA's noise-adaptive capabilities reduce costs without sacrificing performance, achieving comparable or superior results with less advanced LLMs at up to 10 times lower cost. This highlights the potential of LLM-based models in merging cross-domain and cross-lingual KGs. Future work will explore incorporating real-time learning capabilities to dynamically adjust to evolving knowledge bases.
Acknowledgement
The work was fully supported by a grant from the Research Grants Council of the Hong Kong Special Administrative Region, China (Project No. PolyU 15200023).
10

---

References
Rose Catherine and William Cohen. Personalized recommendations using knowledge graphs: A probabilistic logic programming approach. In Proceedings of the 10th ACM conference recommender systems, pp. 325-332, 2016.
Ziwei Chai, Tianjie Zhang, Liang Wu. Kaiqiao Han, Xiaohai Hu, Xuanwen Huang, and Yang Yang. Graphllm: Boosting graph reasoning ability of large language model. arXiv preprint 10.05:23 2023.
Hao Chen, Yuanchen Bei, Qijie Shen, Yue Xu, Sheng Zhou, Wenbing Huang, Feiran Huang, Senzhang Wang, and Xiao Huang. Macro graph neural networks for online billion-scale recommender systems. In Proceedings of the ACM on Web Conference 2024, pp. 3598-3608, 2024a.
Muhao Chen, Yingtao Tian, Mohan Yang, and Carlo Zaniolo. Multilingual knowledge graph embeddings for cross-lingual knowledge alignment. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pp. 1511-1517, 2017.
Shengyuan Chen, Yunfeng Cai, Huang Fang, Xiao Huang, and Mingming Sun. Differentiable neuro- symbolic reasoning on large-scale knowledge graphs. Advances in Neural Information Processing Systems, 36, 2024b.
Shengyuan Chen, Qinggang Zhang, Junnan Dong. Wen Hua, Jiannong Cao, and Xiao Huang. Neuro- symbolic entity alignment via variational inference. arXiv preprint 1rX1v:2410.04153, 2024c.
Zhikai Chen, Haitao Mao, Hongzhi Wen, Haoyu Han, Wei Jin, Haiyang Zhang, Hui Liu, and Jiliang Tang. Label-free node classification on graphs with large language models (LLMs). In The Twelfth International Conference on Learning Representations, 2024d.
Junnan Dong, Qinggang Zhang, Xiao Huang, Keyu Duan, Qiaoyu Tan, and Zhimeng Jiang. Hierarchy- aware multi-hop question answering over knowledge graphs. In Proceedings of the ACM Web Conference 2023, pp. 2519-2527, 2023.
Junnan Dong. Zijin Hong, Yuanchen Bei, Feiran Huang, Xinrun Wang, and Xiao Huang. Clr-bench: Evaluating large language models in college-level reasoning, 2024a. URL abs/2410. 17558.
//tpiv//xvvvv///////////aa//a.aa org/
Junnan Dong, Qinggang Zhang, Chuang Zhou, Hao Chen, Daochen Zha, and Xiao Huang. Cost- efficient knowledge-baser question answering with large language models. arXiv preprint arXiv:2405.17337, 2024b.
Junnan Dong, Qinggang Zhang, Huachi Zhou, Daochen Zha, Pai Zheng, and Xiao Huang. Modality- aware integration with large language models for Enowledge-basee visual question answering, 2024c.
Huang Fang, Yang Liu, Yunfeng Cai, and Mingming Sun. MIn4kb; an efficient markov logic network engine for large-scale knowledge bases and structured logic rules. In Proceedings of the ACM Web Conference 2023, pp. 2423- 2432, 2023.
Fuzhen He, Zhixu Li, Yang Qiang, An Liu, Guanfeng Liu, Pengpeng Zhao, Lei Zhao, Min Zhang, and Zhigang Chen. Unsupervised entity alignment using attribute triples and relation triples. In Database Systems for Advanced Applications: 24th International Conference, DASFAA 2019, Chiang Mai, Thailand, April 22-25, 2019, Proceedings, Part 24. pp. 367-382, 2019.
Feiran Huang, Zefan Wang. Xiao Huang, Yufeng Qian, Zhetao Li, and Hao Chen. Aligning distilla- tion for cold-start item recommendation. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1 1147-1157, 2023.
Xuhui Jiang, Yinghan Shen, Zhichao Shi, Chengjin Xu, Wei Li, Zixuan Li, Jian Guo, Huawei Shen, and Yuanzhuo Wang. Unlocking the power of large language models for entity alignment. arXiv preprint ar 2xX.: 2024.
Ernesto Jiménez-Ruiz and Bernardo Cuenca Grau. Logmap: Logic-based and scalable ontology matching. In The Semantic Web-ISWC 2011: 10th International Semantic Web Conference, Bonn, Germany, October 23-27, 2011, Proceedings, Part I 10, pp. 273-288, 2011.
11

---

Zhixun Li, Xin Sun, Yifan Luo, Yanqiao Zhu, Dingshuo Chen, Yingtao Luo, Xiangxin Zhou, Qiang Liu, Shu Wu, Liang Wang, et al. Gslb: the graph structure learning benchmark. Advances in Neural Information Processing Systems, 36, 2024.
Bing Liu, Harrisen Scells, Wen Hua, Guido Zuccon, Genghong Zhao, and Xia Zhang. Guiding neural entity alignment with compatibility. arXiv preprini arXiv:22/1.158833 2022.
Yang Liu, Huang Fang, Yunfeng Cai, and Mingming Sun. Mquine: a cure for" z-paradox"in knowledge graph embedding models. arXiv preprint arXiv:2402.03358, 2024a.
Yang Liu, Chuan Zhou, Peng Zhang, Yanan Cao, Yongchao Liu, Zhao Li, and Hongyang Chen. Cl4kge: A curriculum learning method for knowledge graph embedding. arXiv preprint arXiv:2408. 14840, 2024b.
Zirui Liu, Chen Shengyuan, Kaixiong Zhou, Daochen Zha, Xiao Huang, and Xia Hu. Rsc: accelerate graph neural networks training via randomized sparse computations. In International Conference on Machine Learning, pp. 21951-211911 PMLR, 2023.
Xin Mao, Wenting Wang, Yuanbin Wu, and Man Lan. Boosting the speed of entity alignment 10x: Dual attention matching network with normalized hard sample mining. In Proceedings of the Web Conference 2021, pp. 821-832, 2021.
Zhiyuan Qi, Ziheng Zhang, Jiaoyan Chen, Xi Chen, Yuejia Xiang, Ningyu Zhang, and Yefeng Zheng. Unsupervised knowledge graph alignment by probabilistic reasoning and semantic embedding. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pp. 2019-2025, 2021.
Meng Qu and Jian Tang. Probabilistic logic neural networks for reasoning. Advances in neural information processing systems, 32, 2019.
Fabian M. Suchanek, Serge Abiteboul, and Pierre Senellart. Paris: Probabilistic alignment of relations, instances, and schema. In Proceedings of the 38th International Conference on Very Large Databases, pp. 157-168, 2012.
Zequn Sun, Wei Hu, Qingheng Zhang, and Yuzhong Qu. Bootstrapping entity alignment with knowledge graph embedding. In IJCAI, number 2018, 2018.
Zequn Sun, Qingheng Zhang, Wei Hu, Chengming Wang, Muhao Chen, Farahnaz Akrami, and Chengkai Li. A benchmarking study of embedding- based entity alignment for knowledge graphs. Proceedings of the VLDB Endowment, 13(1)):2326-22 23400 2020.
Jie Tang, Jing Zhang, Limin Yao, Juanzi Li, Li Zhang, and Zhong Su. Arnetminer: extraction and mining of academic social networks. Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining. pp. 990-998, 2008
Xiaobin Tang, Jing Zhang, Bo Chen, Yang Yang. Hong Chen, and Cuiping Li. Bert-intaa bert- based interaction model for knowledge graph alignment. In Proceedings of the Twenty-Ninhh International Joint Conference on Artificial Intelligence, IJCAI-20, pp. 3174-3180, 2020.
Yu Wang, Nedim Lipka, Ryan Rossi, Alexa Siu, Ruiyi Zhang, and Tyler Derr. Knowledge graph prompting for multi-document question answering. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 19206-19214, 2024.
Zhichun Wang. Qingsong Lv, Xiaohan Lan, and Yu Zhang. Cross-lingudd knowledge graph alignment via graph convolutional networks. In Proceedings of the 2018 conference on empirical methods in natural language processing, pp. 349-357, 2018
Gerhard Weikum and Martin Theobald. From information to knowledge: harvesting entities and relationships from web sources. In Proceedings of the twenty-ninth ACM SIGMOD-SIGACT SIGART symposium on Principles of database systems, pp. 65-76, 2010.
Xuansheng WU, Huachi Zhou, Wenlin Yao, Xiao Huang, and Ninghao Liu. Towards personalized cold-start recommendation with prompts. arXiv preprint arXiv:2306.1 2tli:2306..7566.. 17256, 2023.
12

---

Yuting Wu, Xiao Liu, Yansong Feng, Zheng Wang, Rui Yan, and Dongyan Zhao. Relation Relation-aware aware entity alignment for heterogeneous knowledge graphs. Proceedings of the Twenty-Eight International Joint Conference on Artificial Intelligence, 2019.
Qinggang Zhang, Junnan Dong, Hao Chen, Wentao Li, Feiran Huang, and Xiao Huang. Structure guided large language model for sql generation. arXiv preprint arXiv:2402. 13284, 2024a.
Qinggang Zhang. Junnan Dong, Hao Chen, Daochen Zha, Zailiang Yu, and Xiao Huang. Knowgpt: Knowledge injection for large language models, 2024b.
Yuyu Zhang, Xinshi Chen, Yuan Yang, Arun Ramamurthy, Bo Li, Yuan Qi, and Le Song. Efficient probabilistic logic reasoning with graph neural networks. In International Conference on Learning Representations, 2020.
Jianan Zhao, Le Zhuo, Yikang Shen, Meng Qu, Kai Liu, Michael Bronstein, Zhaocheng Zhu, and Jian Tang. Graphtext: Graph reasoning in text space. arXiv preprint (rxix:2310000000 2023a.
Yu Zhao, Yike Wu, Xiangrui Cai, Ying Zhang, Haiwei Zhang, and Xiaojie Yuan. From alignment to entailment: A unified textual entailment framework for entity alignment. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 8795-8806, 2023b.
Ziyue Zhong, Meihui Zhang, Ju Fan, and Chenxiao Dou. Semantics driven embedding learning for effective entity alignment. In 2022 IEEE 38th International Conference on Data Engineering (ICDE), pp. 2127-2140, 2022.
Huachi Zhou, Jiaqi Fan, Xiao Huang, Ka Ho Li, Zhenyu Tang, and Dahai Yu. refinement by collaborative attributes modeling for click-through rate prediction. In Proceedings of the 31st ACM International Conference on Information Knowledge Management, pp. 4732 47364776, 2022.
13

---

A Notations and algorithms
A.1 Notations
Table 3: Notations.
Notation
Description
G.G'
The source and target knowledge graphs, respectively
E
The sets of entities in G and G", respectively
R. R'
The sets of relations in 9 and G', respectively
T.T'
The sets of relational triplets in G and g", respectively
A
The ground truth set of aligned entity pairs The query budget for the LLM The functionality of relation The alignment probability of entity pair (e, The alignment probability the top-match entity for
B
F(r)
P(e c
P(c) = max P(c=e))
C, L'
The annotated pseudo- label set and the refined pseudo-label set The overall incompatibility of the pseudo-label set The number of iterations of active selection The number iterations of label refinement
(CC
Atr
A.2 Pseudo-code of the greedy algorithm
Below present the pseudo-code of the greedy algorithm, that incorporates probabilistic reasoning to refine the label set.
Algorithm 1 The greedy label refinement algorithm
Inputs: The pseudo-label set L
Parameters: The intialization probability &a (0, the threshold &1 E (do, 1), probabilistic reasoning
iterations nir
Outputs: The refined pscudo-label set L C C
L' + Ø.
We E E,Ve e E', V(e,e') E L.
60
i
while < nu do
Yea E Ee E E, P(en = ehh + 1- 1111-llee (e (APE x
( '(ET"'
(1 (P)P Cr) =d)).
/* Update entity alignment prohahilities.*/
Vr E RR ER, P(r C Σ (1--Il(e, eper P(c) =o P(ee=zai))
/+ Update subrelation
1-1-114 zf (1- P(.
probabilities. +/
L' + C'U e E P(e = = > do} / Label adjustment, add confident pairs to the label set. L' C'' eh) L P(en ex) max (maxees P(e, eh,,, max./e" P(en, /* Label
adjustment, remove less confident pairs from the label set. +/ P(e = max (P(c = e'), for each (c,e/) L' Elevate alignment probability of confident pairs. */
end while
L- + L' U ((e,e'))l(e
81}
/* Augment the refined label with confident pairs. /
Return
A.3 Efficient implementation of label refiner
Parameter-efficien probabilistic reasoning. The total number of alignment probabilities for all entity pairs is |E],"", resulting a large parameter when the KGs involved are extensive. We enhance memory efficiency by adopting lazy inference strategy in probabilistic reasoning. This
14

---

strategy involves only saving the alignment probabilities of the most probable alignments:
{
Plenn Pleneehiben E E E E P(en = = max
max BEE' P(e,e(): max Plen ) e'EB
(12)
Probabilities of other entity pairs can be inferred from these saved alignment probabilities using Eq. (5) when necessary. In this way, parameter complexity is reduced to O (Iε) + [E))))
Computation efficiency of probabilistic reasoning. Probabilistic reasoning is executed iteratively, with each iteration updating the alignment probabilities of all entities and relations following Eq. (5)
and Eq. (6). We detail the analysis of these two update phases in the following separately.
Since we adopt a lazy inference strategy, the update of entity alignment probabilities involves updating the set of most probable alignments and associated probabilities as shown in Eq. (12). This update requires the estimation of all P(e, e & and all P(e,,e'), E E", for each current pair (eh, e))) to determine if this pair needs an update. Consequently, the computational complexity of this process proportional to the size of this set, which is O(EE). The estimated scores P(e,c))
& and P(e,, e'), E E' can be precomputed in advance and reused for all pairs (en, c2), leading to a computation complexity of E(T). Thus, the overall computational complexity for updating entity alignment probabilitiess is O((E|E")| + [&]) O(EE)E"")
The update process for sub-relation probabilities involves updating all P(r C r') for E R and r' E R2, resulting in a complexity of (( The estimation of Eq. (6) utilizes the probabilities of the most probable alignments from Eq. (12). Notably, most relation pairs r') do not have aligned head entities (eA, e/,) or aligned tail entities (er,e)). thus most relation pairs can be excluded for efficient computation by exploiting this sparsity heuristic, reducing the computations by orders.
is worth noting that these computations can be further accelerated through parallelization, as their execution solely depends on the results from the previous iteration.
B Experimental details
B.1 Hardware and software configurations
Our experiments were conducted on a server equipped with six NVIDIA GeForce RTX 3090 GPUs, 48 Intel(R) Xeon(R) Silver 4214R CPUs, and 376GB of host memory. The models were implemented using TensorFlow, NumPy, and SciPy. It was observed that the software version significantly affects hardware-software compatibility. Specifically, the original implementation of Dual-AMN was based on TensorFlow 1.14.0, which is not compatible with newer GPUs such as the NVIDIA GeForce RTX 3090. Therefore, we updated the code be compatible with TensorFlow 2.7.0, enabling the model to leverage GPU acceleration effectively. The details of the software packages used in our experiments are listed in Table 4.
Table 4: Package configurations of experiments.
Package
tqdm
numpy
scipy
tensorflow
keras
openai
Version
4.66.2
1.24.4
1.10.1
2.7.0
2.7.0
1.30.1
B.2
Dataset statistics and preprocessing
In our experiments, utilized the OpenEA dataset version 1.1 (V2), specifically the 15K set. The statisties are detailed in Table 5. It's important to highlight that the destination dataset for D-W-15K. originating from Wikidata, contains only entity IDs, lacking explicit names. These IDs, devoid of semantic content, are not inherently meaningful to a language model. To rectify this, we processed the dataset using the "wiiidttawiki-200 the raw data necessary for constructing the D-W D-W-15K dataset, enabling us to extract meaningful entity names.
aabtt--ttttttttmml dump file from Wikidata. This file provided
We shown the quantity of entities with names (after name extraction) for each KG in the 'Named entities' column in Table 5.
15

---

Table Data statistics of used OpenEA dataset.
KG
Relations
Relation triplets
Attributes
Atribute triples
Entities
Targets top-k:
English (EN)
193
96,318
189
66,899
15,000
EN-FR
13,550
French (FR)
166
80,112
221
68,779
15,000
English (EN)
169
84,867
171
81,988
15,000
EN-DE
13,330
German (DE)
96
92,632
116
186,335
15,000
D-W
DBpedia (DB)
167
73,983
175
66,813
15,000
12,910
Wikidata (WD)
121
83,365
457
175,686
13,458
DBpedia (DB)
72
68,063
90
65.100
15,000
15,000
Yago (YG)
21
60,970
20
131,151
15,000
In the counterpart filtering phase, we selected the top-k (where A: = 20 for our experiments) most similar candidates. The "Target in top-k' column of Table 5 shows the number of target entities included in this selection.
B.3 Performance-cost comparison between GPT-3.55 and GPT-4
Table 6: Performance-cost comparison between GPT-3.5 and GPT-4 as annotator. We increase the budget for GPT-3355 to evaluate its performance. [nx] denotes using nx of the default query budget.
EN-FR-15K
EN-DE-15K
D-W-15K
D-Y-15K
Hive 1
Hie 10
MRR
Hir6:
Hir 10
MRR
Hit@1 1
Hire 10
MRR
Fit@ 1
Hit 10
MRR
GPT:3.5
74.2+0.3
92.9+0.4
81.0.0.3
89.1s0.5
97.8+0.1
92.620.3
87.5+0.3
96.720.1
90.9
97.7+0.0
99.520.0
98. 3+0.0
GPT:3.5 (1.2x)
75.4=0.4
93.2+0.4
81.9+0.2
90.2+0.6
98.0+0.0
93.3+0.4
88.4+0.1
97.1+0.2
91.7=0.1
97.6+0.0
99.3+0.1
98.2+0.0
GPT:3.5 (1.4x)
77.2+0.2
94.5+0.5
83.4.0.3
91.0=0.4
98.1
93.9+0.3
97.9+0.0
92.6-0.1
97.7+0.
99.5=0.0
98.4+0.0
GPT:3.5 (1.6x)
76.3:2.4
94.120.7
82.7&1.9
91.420.2
98.410.0
94.2a0.2
89.630.0
97.920.2
92 810.1
97.7a0.0
99.420.0
98.360.0
GPT-3.5 (1.8x)
78.8:0.5
25.210.4
94.930.5
91.910.1
98.120.0
94.420.1
10.5
97.9:0.2
93. 1:0.4
97.720.
99.520.1
98.310.0
GPT-3.5 (2x)
80.620.2
95.950.1
86.3+0.1
92.7+0.2
98.5+0.1
95.0+0.2
90.750.2
98.5+0.1
93.720.1
97. 97.8+0.0
99.5=0.0
98, 44.00
GPT-4
80.240.3 96.0:0.2 86.0.0.2
98.720.2 95.320.3
89.840.3
97.9.0.2
92.920.3
98.520.1
B.4 Performance comparison against rule-based models
Table 7: Performance comparison against rule-based models, evaluated by precision, recall, and
fl-score in %.
EN-ER-15K Rocall
EN-DE-I5K Recall
D-W-15K Recall
D-Y-15K Recall
Procision
Fl-score
Precision
Fi-score
Precision
Fi-seore
Procision
Fi-scare
Emb-Mateh
88.9
73.6
80.5
89.7
69,5
78.3
91.8
62.4
74,3
100
100
100
Str-Match
69.8
76.6
71.4
80.5
96.2
76.9
PARIS
58.3+0.5
26. 26.5 36.5=0.4
90.8+0.3
50.7+0.3 65.0.0.3
92.4=0.4
70.2+0.2
79,840.2
99.1=0.1
96.7+0.1
97.940.1
LLM4EA
68.6600. 53,1 I+0.2
59.8=0.2
90.5+0.3
82.4+0.4
86.2+0.4
90.7+0.4
81.690,5
85.940.5
98 98.9=0.0 97.6 97.620.1 98.3
In this section, we compare the LLM4EA model with several rule-based models, including two lexical matching- based approaches: Emb-Match and Str-Match. Emb-Match uses cosine similarities between word embeddings to identify aligned pairs, employing pretrained language model. Str- Match utilizes the Levenshtein Distance calculate similarity scores. Additionally, the probabilistic reasoning model PARIS performs entity alignment by relying on probabilistic methods.
For the lexical matching- based models, we compute the similarity and evaluate the confident entity pairs, specifically targeting those whose normalized similarity scores exceed 0.8. The pretrained language model used for Emb-Match is bert-base- uusseed for its ability to process unseen words as a subword model. To reduce false positives, we implementing filtering process that only considers 1-1 matching for embedding- based matching. For PARIS, we assess all inferred aligned pairs by setting the threshold to zero. The entity alignment (EA) model of LLM4EA generates a ranked score list, which is not directly comparable with these rule-based models. To facilitate comparison, we use the trained EA model to generate confident pairs, ensuring that each entity is the top-ranked candidate for its counterpart. These pairs are then processed through our label refiner, and the refined pairs are evaluated. Experiments for PARIS and LLM4EA are repeated three times to ensure statistical reliability; for Emb-Match and Str-Match, experiments are performed once as these algorithms are deterministic. The results are presented in Table 7.
16

---

The findings highlight several key observations: 1) Lexical matching- -aatd methods show degree of alignment ability by leveraging name similarity, particularly on the D-Y-15K dataset, where many aligned entities have identical names. 2) These methods, however, encounter scalability challenges due to their reliance on name similarity. increases, making it difficult to maintain precision. This is evident from our experiments on EN-FR- 100K and WN-DE--O0K, where precision dropped to 49% and recall to 65%, a significant decrease compared to the 15K-size datasets, supporting analysis. 3) PARIS achieves precise results by handling noisy data through probabilistic reasoning, although its recall lower than LLM4EA's. 4) LLM4EA, with its active selection and label refinement techniques, consistently delivers robust and accurate performance across all datasets.
the size of the KG grows, the number of similar entities
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have clearly made the main claims in both the abstract and introduc- tion. Specifically, we have summarized and itemized the contributions at the end of the introduction section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see page 9, Section for the Limitations section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-speciiiaatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
18

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We have provided the code for the framework, accessible via this URL: https: https://github. com/ wwwehhhheeeemmeeeeeee 11m4ea_official.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
19

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Code and datasets are accessible through this anonymous URL: https: / //aonymouu wwwp://mooo.ooooooooooo.....m. science/
In this repository, we included detailed instructions for running the code in the readme. md file.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm ca/
mublic/gudde wttp///oooSuSsiioooooiii
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /nips. hwwpli/gguuueg//////eeeee gwwies//ee/////// http://ggiiieeeoeeuuuuiiiiiiii www.sSubmissisopooooo
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We present the experimental setting details at the beginning of the experiment section. We also provide the dataset statistics and the preprocessing details the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: To control for the randomness introduced by the Large Language Models, repeat each experiment three times and report the mean and standard deviation tables such as Table Table 2, Table 6, and Table 7. For visualized figures such as Figure 2 and Figure 4, we show the error bars representing the standard deviation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have provided the computation resources required reproduce our experi- ment in Appendix B.I.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: https://neut http://eeeuuuuueeeeeeeeeer httph//www....uuup.eeepppee cc/public/ Copublic/Ethie htte/etttiissiieeeeeeeeeeeeeeeee eeGuidelineess
Answer: [Yes]
Justification: We checked and ensured that our paper conforms with the NeurlPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper focuses on technical improvements in entity alignment using large language models and does not introduce direct societal impacts.
Guidelines:
21

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses no such risks, as the work involves only querying the API of LLM to get pseudo-labels, without releasing generative model or datasets.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited the original papers of the datasets and EA models used in our experiments, ensuring proper credit and respect for their licenses and terms of use. For the open-source code and datasets, we have also stated the licenses they use the readme.txt file in our code repository.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
22

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Code of the proposed framework is released and accessible via this anony- mouse URL: https: // //anonyyoou 4open coopen.sciine wcwiece/r/enneeeeeeeeeeeeeeeeeee httpa//////mmeeeeerrr.2222222220 nwwr1ps2024-27733333 It contains readme.md file and GPLv3 license.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
23

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "Xah6ekbOnm", "reviewer": "Reviewer_GZQy", "summary": "The paper proposes an active learning/weak supervision based approach for knowledge base alignment (aligning entities across KGs). The paper explores an LLM labeler for generating entity alignment labels but explores active selection of source nodes labeled by the LLM, a label refiner which refines the LLM annotations based on the structural consistency of the KGs (nodes connected to aligned nodes should be aligned), and finally trains a entity alignment model on the gathered data. The paper compares to a range of baselines and presents several ablations to indicate the efficacy of the proposed approach.", "questions": "NA", "limitations": "NA", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper proposes an interesting approach which seems novel and applied to a meaningful problem.\n- The paper's experiments are thorough.", "weaknesses": "- The papers writing is hard to follow and could benefit from reduction in notation, addition of intuition for modeling choices, and clearer distinguishing of its own contributions wrt prior work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and effort in reviewing our work. We appreciate your insights and will address these points. Below is a summary of the content of our work.\n\n- **Motivations.** Our work is motivated by the fact that existing methods heavily rely on accurate seed alignment pairs for training. Annotating such pairs requires substantial cross-domain knowledge, making it very costly. Large Language Models (LLMs) offer new opportunities in automating this task by processing semantic information and generating pseudo-labels.\n- **Challenges.** Employing LLMs for this task is nontrivial because the search space is vast, and LLMs can generate false labels, which can harm the final performance. There is no existing framework to handle these challenges effectively.\n- **Contributions**. As itemized at the end of the introduction section (lines 69-84), our contributions include 1) A novel LLM-based in-context learning framework for label-free entity alignment; 2) An unsupervised label refiner to enable effective training on noisy pseudo-labels; and 3) An active sampling module to maximize the utility of the annotation budget.\n\nWe sincerely thank you for your recognition and helpful feedback. We will further polish the paper presentation based on your comments in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RftRRoHAEj", "reviewer": "Reviewer_GwVz", "summary": "This paper tackles the challenge of entity alignment (EA) in merging knowledge graphs (KGs) by leveraging Large Language Models (LLMs) to automate annotations, addressing the costly and impractical reliance on human-generated labels despite difficulties like large annotation space and noisy labels. They propose LLM4EA, a unified framework designed to harness LLMs effectively for EA tasks. Key contribution include a novel active learning policy that reduces the annotation space by prioritizing the most valuable entities based on the overall inter-KG and intra-KG structure, and an unsupervised label refiner that enhances label accuracy through probabilistic reasoning. The framework iteratively optimizes its policy using feedback from a base EA model. Experimental results on four benchmark datasets highlight the framework's effectiveness, robustness, and efficiency, showcasing its potential as a significant advancement in the field of entity alignment.", "questions": "Please see the weakness.", "limitations": "This paper has one limitation section.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper proposes an iterative annotation framework leveraging the capability of LLMs' zero-shot in-context learning (ICL). The proposed framework, using an iterative refinement strategy, can effectively reduce costs by utilizing cheaper LLMs such as GPT-3.5.\n- The experimental design of the paper is reasonable and sufficiently demonstrates the effectiveness of the proposed framework.\n- This paper is well-written and easy to follow.", "weaknesses": "- The authors do not consider any open-source LLMs such as Llama2/3. For example, Llama3 8b/70b may have better performance in EA tasks compared to GPT-3.5.\n- The authors did not report the actual API costs for the experiments. For the baselines, what budgets were used to annotate the dataset to train these models? Are they the same as those used in your iterative framework?\n- Figure 1: Replacing the human icon with a robot for the LLM annotator may be better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful feedback that helps us improve the draft. We answer concerns and address the three weaknesses (w1, w2, w3) in the following.\n\n**[w1] Ablation of LLM model**\n\nThe choice of LLM model can affect annotation accuracy and cost. We recognized this factor and employed both GPT-3.5 and GPT-4 as annotators, comparing the differences (Table 1 & Figure 2 in the draft) between less powerful and more powerful LLMs. Thank you for the suggestion; we will include experimental results of additional LLMs, especially open-source ones, to complement the empirical analysis.\n\n\n**[w2] Cost measurement**\n\nAs we have stated in the experimental setting (line244-245), baselines and our framework use the same budget (line239), which is $0.1\\times|\\mathcal{E}|$. To ensure a fair comparison, we use the same prompt template for all experiments. This also ensures that the actual API cost for baselines and our framework is statistically the same (although a slight variance brought by the different lengths of entity names). Based on our experimental observation, the average API cost for annotating each entity is around 1100 tokens, thus it takes around 1100x(0.1x15000)x0.5/10^6 =0.825 dollars to run each alignment task using gpt-3.5-turbo-0125.\n\n\n**[w3] Improvement of presentation.**\n\nThank you for the detailed review, we will replace the icon in figure1 in the draft revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W3EkWDhP1S", "reviewer": "Reviewer_n1nd", "summary": "This paper proposes a new setting that uses LLM to generate entity alignment training pairs, then use the generated pairs to train a matching model for entity alignment between two KGs. The proposed method is evaluated on the OpenEA dataset and achieves better performance than baseline methods on the setting.", "questions": "See above", "limitations": "A limitations section is provided in the paper", "rating": 7, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "A new setting for entity alignment.\n\nThe proposed method achieves better performance than baseline methods on the OpenEA dataset in the setting.\n\nThe paper is well-written and easy to follow.", "weaknesses": "1. The setting is impractical. \nIs LLM really necessary for generating training pairs of entity alignment? if you already know the names of entities in two KGs, you can just use string matching / semantic similarity to generate training pairs. It produces much more accurate training pairs than LLM with \"a fixed budget\". I don't see the point of using LLM for generating training pairs of entity alignment. For the mono-lingual datasets, simply string matching can already achieve 100% accuracy, and for the cross-lingual datasets, lots of papers using semantic similarity have proven its effectiveness (https://arxiv.org/abs/2210.10436, https://aclanthology.org/2022.acl-long.405/, https://www.ijcai.org/proceedings/2020/439, https://dl.acm.org/doi/abs/10.1145/3404835.3462870, https://arxiv.org/abs/2203.01044, https://aclanthology.org/2021.emnlp-main.226/). \n\n\n2. Line 237 \"We have chosen \"V2\" because it more closely resembles existing KGs\". This statement is not true. In V2's generation process, they first randomly delete entities with low degrees, which results in a denser graph. This is not the case for existing real-world KGs, which are usually sparse. In their original paper, they said that the V2 benchmark is \"more similar to existing datasets\" like DBP15k, these datasets are not real-world KGs, they are synthetic datasets. Thus, the statement is misleading and should be corrected.\n\n3. Following the above point, the proposed method is evaluated on the OpenEA V2 dataset, which is twice denser than real-world KGs. The performance on the OpenEA V2 dataset may not be representative of the performance on real-world KGs. This raises the question of the generalization of the proposed method to real-world KGs.\n\n\n4. Baselines are old, see point 1 for more details.\n\n5. The setting can actually be seen as a special case of the setting in https://dl.acm.org/doi/10.1145/3394486.3403268. The difference is the noise's source, in this paper, the noise comes from LLM, in the other paper, the noise is manually added. Mitigating noise from the training set/textual label is not new, they can be found in point 1's references.\n\n6. As far as I know, the DBpedia, Wikidata, and YAGO datasets are all from the same domain, which gives them identity entity names. Take D_Y_15K_V2 for example, if you calculate the edit distance between the entity names in these datasets, you will find that the same entity in different datasets has exactly 0 edit distance. How exactly is Table 7 obtained? Table 7 shows the performance of semantic similarity/string matching is a lot worse than the proposed method and the F1 can be as low as 2% in some cases. This is not true, the F1 of semantic similarity/string matching should be 100% in the monolingual case.  The results in Table 7 are not reliable and should be corrected. I suggest the authors to check the correctness of the dataset and the code implementation.\n\nOverall, this paper contains interesting but impractical new settings, false statements, and unreliable results. I recommend rejection.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We respectfully thank you for the helpful feedback that helps improve the draft. We answer the concerns and address weaknesses (w1-w6) as below:\n\n# [w1, w4] Reasonability of using LLMs in EA and comparison with recent methods\n\nWe sincerely thank for pointing out this subtle point, and we answer this question from following two aspects:\n\n1.   **Potential in more challenging scenario.** Although we perform evaluation in a simple scenatio (entity names are available) in the experiments, the potential of using LLMs in EA tasks is more than that. Specifically, **language models can annotate challenging datasets where no entities names are available**, by exploiting the semantic information within textual attributes, where **existing name-based methods (e.g. LightEA, SEU, BERT-INT) cannot handle**. To empirically evaluate this, we entity alignment on D_W_15K_V1 with LLM4EA, where the target entities are IDs to avoid name bias. Results are as below, \n\n     | Alignment performance | Annotation precision    |\n     | -- | - |\n     | 0.472 (hit@1) | 301 (true positive) |\n     | 0.654 (hit@10)   | 49 (false postive)  |\n     | 0.536 (MRR)  | 1150 (abandoned)  |\n\n     As shown in the results, LLMs can generate effective pseudo-labels without the need for identical names and get promising results. \n\n2.   **Scalability to larger KGs.** The target of entity alignment is building a larger unified KG. Although existing finetuning-based models (BERT-INT, SDEA, CEA) can also leverage semantic information, our incontext-learning based framework 1) does not require pre-aligned labels to finetune; and 2) is more scalable as it does not require local hardware configuration for LLMs.\n\nwe appreciate the listed methods for boosting the comprehensiveness of the discussion of related work, and we will add these results in the draft revision.\n\n# [w2 & w3] Results on OpenEA V1 \n\nThank you for detailed reviews, we will correct the false statement about dataset version in the revision. To validate the performance on real-world KGs, we attach the experimental results on OpenEA V1 dataset as below. Due to character limit, we only show the results of strong&new baselines. \n\n|    | D_W  | (w/o ent  | name)  |  | D_Y  |   |   | EN_DE |  |  | EN_FR  |  |\n| -- | - | -- | - | - | - | - | - | -- | -- | --- | - | - |\n|     | hit@1     | hit@10    | mrr       | hit@1     | hit@10    | mrr    | hit@1     | hit@10    | mrr       | hit@1     | hit@10    | mrr       |\n| BootEA     | 0.302     | 0.584 | 0.413     | 0.664   | 0.811  | 0.718     | 0.655     | 0.844     | 0.720     | 0.429     | 0.685     | 0.515     |\n| Dual-Amn    | 0.386     | 0.639 | 0.479     | 0.748   | 0.858  | 0.790     | 0.728     | 0.910     | 0.793     | 0.537     | 0.800     | 0.628     |\n| RDGCN    | 0.337     | 0.436  | 0.372     | 0.817  | 0.923  | 0.858     | 0.646     | 0.775     | 0.692     | 0.597     | 0.742     | 0.649     |\n| LLM4EA    | **0.472** | **0.654** | **0.536** | 0.755  | 0.859     | 0.795     | **0.736** | **0.918** | **0.800** | 0.581     | **0.819** | 0.662     |\n| LLM4EA (RDGCN) | 0.421  | 0.532  | 0.476  | **0.842** | **0.931** | **0.883** | 0.672     | 0.787     | 0.736     | **0.642** | 0.761     | **0.683** |\n\n\n\nAs shown by the results, LLM4EA consistently outperform strong baselines. It is noteworthy that LLM4EA is a general framework and can employ any base EA model such as RDGCN (last row) to perform robust label-free entity alignment with LLMs.  We argue that **the conclusions and claims holds on this sparse dataset**. And we will include the updated results in the draft revision.\n\n# [w5] Comparison with existing work that addresses noise labels (REA)\n\nThe paper referenced in weakness 5 (REA) also addresses noise in the training set. However, our work is significantly distinct from theirs in three key aspects: **1) Technical novelty**. The noise detection model in REA is trained as a binary classifier using an adversarial training paradigm, In contrast, our label refiner employs probabilistic reasoning for robust label refinement. **2) Flexibility.** LLM4EA can leverage any off-the-shelf EA models as base models without altering the model architecture or training objectives. **3) Superior performance**. We empirically tested the performance of REA on annotated pseudo-labels generated by LLMs and present the results below and show the superiority of LLM4EA.\n\n|        |          D_W       | (w/o ent  | name)            |           | D_Y       |           |           | EN_DE     |           |           | EN_FR     |           |\n| ------ | ---- | ----- | --------- | --------- | ------ | ------ | --------- | --------- | --------- | --------- | --------- | --------- |\n|        | hit@1     | hit@10             | mrr       | hit@1     | hit@10    | mrr       | hit@1     | hit@10    | mrr       | hit@1     | hit@10    | mrr       |\n| REA    | 0.213     | 0.467              | 0.298     | 0.447     | 0.798     | 0.563     | 0.363     | 0.742     | 0.475     | 0.226     | 0.536     | 0.352     |\n| LLM4EA | **0.472** | **0.654**          | **0.536** | **0.755** | **0.859** | **0.795** | **0.736** | **0.918** | **0.800** | **0.581** | **0.819** | **0.662** |\n\n\n\n# [w6] Low recall/f1 on semantic matching\n\nThank you for the detailed review and constructive suggestions. We identified that the low recall was due to the word embedding model not generating embeddings for out-of-vocabulary names. We addressed this by replacing it with a pretrained language model (bert-base-uncased) and implementing a filtering process that only considers 1-1 matching. The updated results are as follows:\n\n|       | Precision | Recall | F1    |\n| ----- | --------- | ------ | ----- |\n| D-W   | 0.918     | 0.624  | 0.743 |\n| D-Y   | 1.0       | 1.0    | 1.0   |\n| EN-DE | 0.897     | 0.695  | 0.783 |\n| EN-FR | 0.889     | 0.736  | 0.805 |\n\nWe hope our response can satisfactorily address your concerns and we would appreciate if you could consider raising the score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6ggsAtPpnz", "reviewer": "Reviewer_SZjY", "summary": "The paper proposes LLM4EA for annotating entity alignment pairs using an LLM. It introduces an active learning policy and an unsupervised label refiner to efficiently collect pseudo-labels. Experiments on OpenEA benchmarks demonstrate the strong performance of LLM4EA.", "questions": "Please see Weaknesses.", "limitations": "N/A", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "The paper is well-structured and straightforward, making it easy to understand the design and motivation behind LLM4EA.\n\nApplying an LLM for label annotation is both simple and effective.\n\nThe results on OpenEA-V2 significantly outperform the baseline methods.", "weaknesses": "The algorithm in this paper somewhat echoes that of BootEA, which limits its novelty. While earlier iterative or bootstrapping algorithms generated pseudo-labels using cosine similarity or other metrics, this paper introduces the use of an LLM to select pseudo-labels. Beyond this aspect, the procedure closely resembles existing methods.\n\nThe declaration in Line 237 is incorrect; \"V1\" is the dataset that resembles real-world KGs, and \"V2\" is the easier version.\n\nTable 1 is not clear. Is LLM4EA the only method evaluated that considers pseudo-labels generated by an LLM? If so, why are the performances of the other methods significantly lower than those reported in the OpenEA paper?\n\nUtilizing LLMs to annotate labels carries a risk of data leakage, as many entity name pairs in the OpenEA datasets are identical. This flaw was also noted by the creators of OpenEA.\nIf the authors account for the entity name as a feature, they should also benchmark LLM4EA against approaches that incorporate textual information, for example, MultiKE.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback, which has greatly helped improve the draft. Below, we address your concerns and weaknesses (w1-w4).\n\n# [w1] Technical Novelty (distinctions from bootEA)\n\nWe respectfully point out that the LLM4EA is significantly distinct from bootEA, we outline three major differences between them:\n\n1. **Pseudo-label generation.** Comparing with the bootEA, our work generates pseudo labels by directly processing semantic information, while bootEA generates pseudo labels based **on similarity scores grounded on embeddings trained on pre-aligned labels**. LLM4EA exploits the potential of automating the EA task in a **label-free** fashion.\n2. **Unsupervised label refinement**. Our label refiner is unsupervised, while BootEA relies on the embeddings **learned on training labels** to compute the label’s confidence. As a result, **directly adopting BootEA to this task cannot mitigate the initial noisy annotated pseudo-labels**. As we have shown in Table 1 and discussed in line 265-268 in the paper draft, BootEA still suffers from false labels if directly trained on the noisy labels from LLMs.\n3. **Optimized resource allocation**. We introduce an active search policy to improve the utility of the query budget.\n\n\n\n# [w2 ] Results on OpenEA V1\n\nThank you for the detailed review and helpful feedback, we will correct the false statement of the dataset version in the revision. To validate the performance on real-world KGs, we attach the experimental results on OpenEA V1 dataset as below. Due to character limit, we only attach results of strong baselines.\n\n|                | D_W       | (w/o ent  | name)     |           | D_Y       |           |           | EN_DE     |           |           | EN_FR     |           |\n| --- | -- | - | -- | -- | -- | -- | -- | -- | -- | -- | -- | - |\n|      | hit@1     | hit@10    | mrr       | hit@1     | hit@10    | mrr       | hit@1     | hit@10    | mrr       | hit@1     | hit@10    | mrr       |\n| BootEA         | 0.302     | 0.584     | 0.413     | 0.664     | 0.811     | 0.718     | 0.655     | 0.844     | 0.720     | 0.429     | 0.685     | 0.515     |\n| Dual-Amn       | 0.386     | 0.639     | 0.479     | 0.748     | 0.858     | 0.790     | 0.728     | 0.910     | 0.793     | 0.537     | 0.800     | 0.628     |\n| RDGCN          | 0.337     | 0.436     | 0.372     | 0.817     | 0.923     | 0.858     | 0.646     | 0.775     | 0.692     | 0.597     | 0.742     | 0.649     |\n| LLM4EA         | **0.472** | **0.654** | **0.536** | 0.755     | 0.859     | 0.795     | **0.736** | **0.918** | **0.800** | 0.581     | **0.819** | 0.662     |\n| LLM4EA (RDGCN) | 0.421     | 0.532     | 0.476     | **0.842** | **0.931** | **0.883** | 0.672     | 0.787     | 0.736     | **0.642** | 0.761     | **0.683** |\n\n\n\n\n\nAs the results show, our model consistently outperforms baselines. Noteworthy that LLM4EA is a general framework and can employ any base EA model such as RDGCN (last row) to perform label-free entity alignment. **We kindly argue that our conclusions and claims hold on this more realistic dataset**. And we will include the updated results in the draft revision.\n\n# [w3] Clarification of baseline setting\n\nAllow us to clarify the confusion about the experimental setting in weakness3.\n\n1.   **The input of baselines are pseudo labels**. \n\nAs stated in the experimental setting (line244-245 in the draft), baseline models are also trained on the pseudo-labels generated by the LLMs. This setting ensures **fair comparison** because the inputs are the same (same query budget to the LLMs). \n\n\n\n2.   **Baselines perform lower than OpenEA's paper for two reasons:**\n\n**1) the annotated pseudo-labels are noisy** and existing methods cannot handle without a pre-refinement of noisy labels before training; and **2) the training label size is smaller** ($0.1|\\mathcal{E}|$, but in OpenEA's paper it's $0.2|\\mathcal{E}|$ ).\n\n# [w4] name bias concern \n\nThe use of name information is often referred to as name bias. We answer this concern from two aspects:\n\n1.   **When name information is used**: LLM4EA can generate pseudo-labels and enable robust learning. While most existing methods like MultiKE and RDGCN exploit entity names as features, they <u>rely on pre-aligned pairs</u> for training. LLMs can serve as pseudo-label generators. \n2.   **When names are not available**: LLM4EA still works. By processing semantics within attributes, LLM4EA can generate effective labels. We empirically evaluate this on D_W_15K_V1 (**1st column** in the following table), we masked target entity names with IDs to avoid name bias.  And compare with MultiKE and RDGCN which both leverage semantic features.\n\n|    | D_W       | (w/o ent  | name)  |    | D_Y   |   |    | EN_DE  |   |   | EN_FR   |    |\n| - | - | - | - | - | - | - | - | -- | - | - | -- | -- |\n|      | hit@1     | hit@10    | MRR     | hit@1     | hit@10    | MRR       | hit@1     | hit@10    | MRR | hit@1  | hit@10    | MRR       |\n| MultiKE  | 0.021     | 0.064   | 0.036     | 0.503     | 0.767     | 0.598     | 0.357     | 0.666     | 0.461     | 0.274     | 0.590     | 0.380     |\n| RDGCN   | 0.337     | 0.436   | 0.372     | 0.817     | 0.923     | 0.858     | 0.646     | 0.775     | 0.692     | 0.597     | 0.742     | 0.649     |\n| LLM4EA    | **0.472** | **0.654** | **0.536** | 0.755     | 0.859     | 0.795     | **0.736** | **0.918** | **0.800** | 0.581     | **0.819** | 0.662     |\n| LLM4EA (RDGCN) | 0.421  | 0.532     | 0.476     | **0.842** | **0.931** | **0.883** | 0.672     | 0.787     | 0.736     | **0.642** | 0.761     | **0.683** |\n\nAs shown in the above table, \n\n-   LLMs serve as an effective annotator for all methods to perform label-free EA, which justifies the above 1st argument.\n-   LLMs can generate effective labels without name information, which justifies the 2nd claim.\n\nWe hope our responses can satisfactorily address your concerns and we would appreciate it if you could consider raising the score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a way to use LLM for Entity Alignment tasks (LLM2EA). It introduced an active learning policy to help reduce the annotation space, which is a unique challenge to vanilla LLMs in EA tasks. It also adds an unsupervised label refiner to efficiently collect pseudo-labels. Experiments showed clear improvement over other methods, including the additional ones suggested during rebuttal.

A main concern about the paper is around novelty (Reviewer SZjY), that it's similar to the core idea of previous methods like BootEA. The authors provided new evaluation data to show that LLM2EA has significantly better results than BootEA. AC acknowledges the concern of SZjY, however given the delta in experiment metrics and the differences explained by the authors, the delta seems non-trivial.
Decision: Accept (poster)