Explore the Way: Exploring Reasoning Path by Bridging Entities for Effective Cross-Document Relation Extraction
Authors: Junyoung Son
Jinsung Kim
Jungwoo Lim
Yoonna Jang
Heuiseok Lim
Venue: EMNLP 2023 Findings (2023)
Keywords: Cross-document relation extraction
Document relation extraction
Reasoning Path Construction

Abstract: Cross-document relation extraction (CodRED) task aims to infer the relation between two entities mentioned in different documents within a reasoning path. Previous studies have concentrated on merely capturing implicit relations between the entities. However, humans usually utilize explicit information chains such as hyperlinks or additional searches to find the relations between two entities. Inspired by this, we propose Path wIth expLOraTion (PILOT) that provides the enhanced reasoning path by exploring the explicit clue information within the documents. PILOT finds the bridging entities which directly guide the paths between the entities and then employs them as stepstones to navigate desirable paths. We show that models with PILOT outperform the baselines in the CodRED task. Furthermore, we offer a variety of analyses to verify the validity of the reasoning paths constructed through PILOT, including evaluations using large language models such as ChatGPT.

Explore the Way: Exploring Reasoning Path by Bridging Entities for Effective Cross-Document Relation Extraction
Junyoung Son, Jinsung Kim, Jungwoo Lim, Yoonna Jang, Heuiseok Lim' Computer Science and Engineering, Korea University
Republic of Korea
(senn,,iin2304,, wjddn803, (mmsseok)kkore. ac.
Abstract
Record label
Jun Jin
SM Entertainment
Cross-document relation extraction (CodRED) task aims to infer the relation between enti- ties mentioned in different documents within reasoning path. Previous studies have concen- trated on merely capturing implicit relations between the entities. However, humans usually utilize explicit information chains such as hy- perlinks or additional searches to find the rela- tions between two entities. Inspired by this, we propose Path wIth expLOraTiion (PILOT) that provides the enhanced reasoning path by ex- ploring the explicit clue information within documents. PILOT finds the bridging entities that directly guide the model with paths be- tween the given entities and then employs them stepstones navigate desirable paths. We show that models with PILOT outperform the baselines in the CodRED task. Furthermore, we provide variety of analyses to verify the valid- ity of the reasoning paths constructed through PILOT, including evaluations using large lan- guage models such as ChatGPT.
Title: Lee Styoung (d*)
Tide: F[+) (group) (d")
But mainstream popularity when the joined the cas and was wnth
lorsttl scouted SM Entertalnment
in 2000 while on famil vist Korez..
Shinlweg Called the Cundam couple
subsequentty appeared is s'salll role Shithing's music vidio song Wailinng Narrh.
mutual love aniiee...
shinkwa (tP))
SM Ennerzainmen sought to releaze
smother group captiallus the
formers Complsed nemberrs
Mun Eric. Kim wan.gwnn.uuu Iif) Shit
Mtye-Sunng Anpy group narret (arnherr
Figure I: An illustrated example of d' and dl being linked by bridging entity Shinhwa. The previous reason- ing path infers the relation between l and & by using only dh and d', but our method constructs the reasoning chain by utilizing the bridging entity Shinhwa.
To implement the cross-document RE model, Wang et al. (2022) has enhanced the method of cap- turing bridging entities by entity-based document- context filter and utilizes cross-document entity relation attention module for enhanced connection awareness between the reasoning paths. On the contrary, Lu al. (2022) focuses on extracting evi- dence path and ranking with dense retrieval with- out explicit consideration for the bridging entity. Likewise, previous studies have only attempted to model the bridging entities implicitly, neglecting to exploit the potentially valuable additional infor- mation that bridging entities can provide.
1 Introduction
The relation extraction (RE) task aims to predict re- lations between two entities in given text and is an essential basis for application tasks such as knowl- edge base construction (KBC) and question answer- ing (QA) (Swampillai and Stevenson, 2010; Ji et al., 2010; Son et al., 2022). Although the sentence-level and document-level RE have achieved substantial performance (Zhang et al., 2017; Yao et al., 2019; Yang et al., 2021: Zhang et al., 2021), it is still challenging to identify the relations across multiple documents. Since the relations between the entities can be inferred from the multiple documents in the wild, cross documen relation extraction (CodRED) is suggested by Yao al. (2021) which provides the relation between two entities (i.e., head and tail) across the multiple documents (Wu et al., 2023).'
However, utilizing the explicit information rele- vant to bridging entities, which can provide defini- tive guide in the exploration of reasoning paths, is crucial for identifying the correct relation. When inferring relations between entities using multiple documents, humans may not follow arbitrary in- IIn the vast knowledge repository of Wikidata, more than 57.6% of the facts are not encapsulated within single
*Corresponding author
Wikipedia documents but are instead distributed across multi- ple documents.

---

formation. Instead, they instinctively draw upon explicitly linked information from the bridging en- tity between documents that which connects a path to the tail entity and provides structural clues about the relation.
2.2 PILOT
2.2.1 Bridging Document Retrieval
Eawaey- Retriever To find the relevant doc- uments De = [de] when given entity as a query, we train a dense retriever (Karpukhin et al., 2020). We first collect the entities which have their corresponding Wikipedia page from CodRED and Wikidata construct the dataset for training the re- triever. Afterward, we regard the Wikipedia page of the entity e as a positive example p and Wikipedia pages extracted from BM25 following Karpukhin et al. (2020) as the negative examples {pr 1/-11- The training objective follows contrastive learn- ing (Chen et al., 2020), shown in Equation 1.
In this paper, we introduce Path wIth expLOraTTion (PILOT), an effective way of con- structing explicit and structural reasoning path uti- lizing bridging entities. According to PILOT, first finds the bridging entities with structural entity information, such as intra- ditra-document relation, and retrieves the documents that are related to the bridg- ing entities. Afterward, we rerank the documents by entity-aware scoring function and leverage the high- scored documents as the stepstones connecting the head and tail. The RE models with PILOT showed consistent improvement in performance compared to the baselines in the CodRED task. In addition, we provide an analysis with a large language model (LLM) such as ChatGPT (OpenAI-Blog. 2022) on whether our enhanced reasoning path leads to more knowledgeable and relevant information.
exp exprimteez"")
(1)
-log expsim(e,pT)) + E/= exp exp sim(c
Bridging Candidate Construction To build a set of candidate bridging entities between two documents, we first utilize the shared entity set Eeand E U E where (E", E²) is the set of all entities that exist in (d (d",d") d') respectively. With the dense retriever, we retrieve a set of documents Deand [d D
2
Method
Our proposed method revolves around leveraging the most significant bridging entities to construct a reasoning path between the head and tail entities. Initially, we employ structural information (i.e., intra-document relations) and the dense retriever assemble a set of candidate bridging entities and their corresponding documents. Subsequently, we select the most relevant among these candidates through entity-based filtering and scoring methods. Finally, we utilize the final bridging entities to con- struct an expanded reasoning path.
by using coand Eeand
i==1
as query.
Entity-based Filtering After retrieving the set of documents, we filter out the documents to preserve more relevant documents to the coand In detail, we exclude drand if @ is not mentioned in drand We further validate goand exploiting structural in- formation from Wikidata. If the e is connected to the h and according to the Wikidata, we use them as a bridging entity candidate or, otherwise. filter out.
2.1 Preliminaries
Task Definition In cross-document RE, head entity and a tail entity are denoted as h, and f, respectively. P [ is set of reasoning paths, and each reasoning path p; has two docu- ments (df; d)), including h and f. The task aims to infer the relation r E R. between two entities where R is a pre-defined relation set. If particular entity is mentioned in both de and d'. it can con- sider this entity According to Yao et al. (2021), There are potential bridging entities that establish the relation between the documents (4.7 on average). We denote a set of the bridging entities as E {el L between two documents.
Bridging Entity Scoring To quantify the rele- vance of candidate entities linked to the head and tail entities, we score each entity based on the given document. In detail, given an entity cand and the document d, we find the entities in the and count the number of occurrences of condd follows:
bridging entity between them.
N ( rand dd = Σ 8(aand ,e),
(2)
eEE2
where 5(c1; 62) returns if cj = 02 otherwise 0. Based on this score, we get the top-k bridging enti- ties E and employ them to construct a path.

---

Development
Test
Method
AUC (r)
FI (o)
P@500 (r)
P@ 1000 (r)
AUC (o)
F1 (a)
End-to-End (Yao et al., 2021)
47.94
51.26
62.80
51.00
47.46
51,02
PILOT
53.23 (0.59)
56. (0.70)
70.86 (0.92)
55.57 (0.25)
54.31 (1.23)
57.33 (1.31)
+ PILOT (- WWikidata)
52,98 (0.98)
55.72 (0.24)
72.72 (0.80)
55.60 (0.27)
53.37 (0.53)
57.52 (1.01)
ECRIM (Wang al., 2022)
60.91
61.12
78.89
60.17
60.67
62,48
+ PILOT
63.83 (7.06)
63,30 (0.35)
79.48 (1.70)
62,54 (0.35)
62,90 (0.96)
63.86 (1.01)
PILOT (- Wikiddtta
63.31 (0.56) 62.53 (0.47)
78.54 (7.45)
62.24 (0.67)
62.19 (7.05)
61.75 (0.74)
Table Comparisons of performances with the baselines on CodRED. ECRIM (Wang et al.. 2022) is the existing state-of-the-art model. Our test results are obtained from the official website of CodRED on Codalab.
performance. Specifically, we witnessed accuracy improvements of up to 3.54% the development set and 2.23% in the test set, validating the broad applicability of PILOT. When Wikidata filtering is removed, the performance decreases 0.4% for the End-to-End and 0.52% for the ECRIM in the de- velopment set, respectively. Especially in ECRIM, the fl score drops 1.43% in the test set. These sults suggest the central role of Wikidata filtering PILOT, underlining the importance of structured information in navigating relational information for cross-document reasoning.
I(eaand) (N(caand d²)) + [(, drand)) (N(e d3) (+++ d(
(3)
Based on this score, we get the top-k bridging entities Ebh and employ them construct a path.
2.2.2 Path Construction
Finally, we construct an expanded reasoning path p' by employing the existing reasoning path P { and EY- = / The detailed equation constructing p' is as follows:
3.2 Experimental Results for Entity-aware Retriever
IP| 1
p' (U 1 P], (4)
=11=1
Development R
Test R@5
Method
Hits@1
MRR
Hits@9
MRR
where d is document retrieved by Entity-aware retriever using query as eij.
DR (dist)
89.28
97.43
92.90
88.92
97.33
92.70
DR (train)
94. 10
98.59
96. 16
93.68
98.47
95 90
DR + L.)
94.41
98.97
96.50
94,03
98.69
96.18
3 Experiments and Analysis
Table Retrieval performances of Entity- aware Re- triever the development and test sets. Both 'dist' and 'd'' refer to the distant supervision manner.
We apply PILOT to the existing reasoning path pro- vided by (Yao et al., 2021) to construct extended reasoning path. For the experiment, we utilize End- to-End (Yao al., 2021) and ECRIM (Wang et al., 2022) models as baselines. The detailed experimen- tal settings are described in 4.
The experimental results of the Entity- Re- triever used for bridging document retrieval in PI- LOT are presented in Table 2. This process sig- nificant since retrieving documents with low rele- vance to the entity could lead to a substantial per- formance decrease, creating a potential bottleneck. The Entity-aware Retriever, with an accuracy of 94.41% % and 94.03%, hints at its pivotal role in PILOT, suggesting its ability provide relevant bridging context the bridging entities.
3.1 Experimental Results in CodRED
We show the experimental results of PILOT method on the CodRED task in Table These re- sults demonstrated a consistent enhancement in the performance across all baselines when they were incorporated with PILOT method. In the End-to- End baseline, PILOT significantly improves per- formance across both the development and test sets. In particular, the substantial performance gain with 6.5% in the test set shows the generalization ability our method. Even with the state-of-the-a model ECRIM, we observed a consistent improvement in
3.3 Evaluating the Capability to Distinguish Correct Reasoning Paths
We evaluate path-level accuracy rather than entity lair-level to validate the proposed method's capa- bility to distinguish correct reasoning paths despite

---

extremely noisy settings. Because CodRED con- sists of only 6.7% of positive reasoning paths and the remaining 93.3% of N/A reasoning paths, we assume that the path-level evaluation can be used to estimate this ability. As shown Table 3, we can
we Uniiping
of mebra
Rassom (hares)
HLDT.
Method
Path-level AUC
End-to-End (Yao et al., 2021)
79.24
Figure 3: Achievement results on Fl by the type of scoring module to select the bridging entity.
+ PILOTT
80.31 (+1.07)
Table 3: Results of Path-level Accuracy on the develop- ment set.
the set of entities that appear in (dh d'). 'Shared entities' the setting where the bridging entities are randomly selected from the set of entities com- monly appearing in d' and d'.
observe that the path-level accuracy improves when the bridged reasoning paths expanded by PILOT are applied to the original reasoning paths despite their highly noised setting.
As shown in Figure 3, the reasoning path con- structed by PILOT shows the most outstanding performance. The performance of random entities ( ('All entities') is significantly worse than when the bridging entity
3.4 Quantity of Bridging Entities Explored
As shown in Figure 2. the RE performance ini- tially improves with more bridging entities. How- ever, as the number of these entities continues to rise, a performance drop is observed. This suggests that using too many bridging entities can introduce more noise, leading to decreased effectiveness. In addition, the performance greatly drops when the number of bridging entities is 5. This result reflects the fact of exceeding the average number of po- tential bridging entities (4.7 on average (Yao et al.,
utilized. It indicates that the performance improvement is not simply due to an increase the amount of information in the bridg- ing entity, but rather to the importance of finding genuinely relevant information.
Lose
Proportion (S1
60-
82.48%
79.85%
82.73%
39..50%
76.22%
81.35%
2021)).
40-
20-
13.25%
54
Out Shared
Owt Shured
Shanei
Outs
Oues
12 (Dee))
Figure 4: Win vs. Lose results (%) evaluated by Chat- GPT between the paths randomly chosen and construed by PILOT
46
Number bridges
3.6 ChatGPT Evaluation
Figure 2: Results on Fl score with respect the number of bridging entities provided per path.
To show the efficacy of our reasoning path from PI- LOT method, we ask ChatGPT to evaluate between the cases where the path is explored by PILOT and the path is constructed with random entities." The evaluation criteria are as follows: 1) Informative- ness indicates how many clucs the provided bridg- ing entity information contains to infer a relation, and 2) Relevancy indicates how much relevant the
3.5 Bridging Entity Type Selection
We demonstrate the cross-document RE perfor- mance based on the type of score function in Fig- ure 3. We compare four settings. "w/o bridging' is the setting where only (d",'") dd') is given as a reason- ing path. 'All entities' indicates the setting where the bridging entities are randomly selected from
"The evaluation prompt template using ChatGPT is illus- trated in Appendix 8

---

given reasoning path, including the bridging entity, is with the given pair of entities (head, tail). Figure illustrates that our reasoning paths got higher win proportion compared to both types of random paths by 64.96% and 65.46% respectively, and this tendency is also similar in relevancy. In addition, Ours All has a higher win rate than Shared vs. All. These results imply that our reasoning paths with PILOT method include more information for relation inference and act as the relevant evidence explain the relation between head and tail.
Limitations
Because PILOT constructs expanded version of the reasoning path based on the given reasoning path which is a document pair (d" d') for h and entities, it requires initial reasoning path to build an expanded one. This suggests that regardless of how well the bridging context is selected, the per- formance could be compromised if the quality of the initial reasoning path is not adequate, imply- ing a dependency on the initial path. This factor should be taken into account when building paths in the open-settings in the future. In addition, the scoring function in PILOT is calculated using the number of mentions in each document, neglecting polentially important context which is infrequent but has a critical role between the entities. As Table 7 demonstrated, the bridging entities chosen from PILOT between entities can be nothing when the retrieved and filtered results return nothing. In other words, while this approach effectively filters out some irrelevant paths, it also carries the potential downside of losing valuable information that could contribute inferring the relations between the entities.
3.7 Human Evaluation
Informativeness
Relevancy
Win
Tie
Lose
Win
Tie
Lose
Ours vs. Shared 55
28
7
53
30
7
vs.
61
27
2
29
Table 4: Results of Human Evaluation.
To support the results mentioned in the Chat- GPT Evaluation, conducted human evaluation by employing three individuals. More specifically, we presented each participant with 30 text snip- pets extracted from randomly sampled reasoning paths and had them undergo an AB test in compari- son with the random baselines. As shown in Table 4, the results of the human evaluation exhibited a trend similar to that of the ChatGPT Evaluation. Notably, Our method demonstrated win rate of 61.1% % against Shared and 67.7% against All in In- formativeness. In terms of Relevancy, it showed win rate of 58.9% versus Shared and 65.5% against All. Additionally, the lower Lose rate compar- ison to the Win and Tie ratios suggests that our methodology has significantly enhanced the qual- ity of the reasoning path.
Acknowledgements
This research was supported by the MSIitMinnstry of Science and ICT), Korea, under the ITRC(hnformation Technology Research Center) support
supervised by the IITP(Institute for Information & Communications Technology Planning & Evaluation). This work was supported by Institute of Information & communications Technology Planning & Evaluation(IITPP grant funded by the Korea goVernmen((MSITT) (No. 2020-0-00368, A Neural-Symbolic Model for Knowledge Acquisition and Inference Techniques). Also, this research was supported by Basic Science Research Program through the National Research Foundation of Korca(NRF) funded by the Ministry of Eddcettion/MRF-2202IRAAAA A03045425).
4 Conclusion
We proposed PILOT, which the method that ex- plores the reasoning path by utilizing the bridging entities and their documents for the CodRED task. Based on the filtering and scoring, PILOT exploits the explicit and structural information from meta- data such as Wikidata, and constructs the enhanced reasoning path. In the experiments, the models with PILOT showed improvements compared to the baselines and outperformed the existing state-of- the-art model. Moreover, we provided additional experiments and extensive analyses to validate the efficacy of explored reasoning path and analysis.
References
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In In- ternational conference machine learning, pages 1597-1607, PMLR.
Heng Ji, Ralph Grishman, Hoa Trang Dang, Kira Griffitt,

---

and Joe Ellis. 2010. Overview of the tac 2010 knowl- edge base population track. In Third text analysis conference (TAC 2010), volume 3, pages 3-3.
Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019. Docred: A large-scale document-level relation extraction dataset. In Pro- ceedings of the 57th Annual Meeting of the Associa- tion for Computational Linguistics, pages 764-777.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online. Association for Computational Linguistics.
Ningyu Zhang, Xiang Chen, Xin Xie, Shumin Deng, Chuanqi Tan, Mosha Chen, Fei Huang, Luo Si, and Huajun Chen. 2021. Document-level relation extrac- tion as semantic segmentation. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 3999-4006. Interna- tional Joint Conferences on Artificial Intelligence Organization. Main Track.
Keming Lu, I Hsu, Wenxuan Zhou, Mingyu Derek Ma,
Muhao Chen, et al. 2022. Multi-hop evidence re- trieval for cross-document relation extraction. arXiv preprint 12.
Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Position-awas aware attention and supervised data improve slot filling. In Proceedings of the 2017 Conference on Empiri- cal Methods in Natural Language Processing, pages 35-45, Copenhagen, Denmark. Association for Com- putational Linguistics.
OpenAI-Blog. 2022. Chatgpt: Optimizing language models for dialogue.
Junyoung Son, Jinsung Kim, Jungwoo Lim, and Heuiseok Lim. 2022. GRASP: Guiding model with RelAtional semanties using prompt for dialogue re- lation extraction. In Proceedings of the 29th Inter- national Conference on Computational Linguistics, pages 412-423, Gyeongju. Republic of Korea. Inter- national Committee on Computational Linguistics.
Appendix
A Experimental Setup
A.1 Hyperparameters
Kumutha Swampillai and Mark Stevenson. 2010. Inter- sentential relations in information extraction cor- pora. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (0)
We conduct our experiments using the closed- setting of CodRED, We employ the cased version of BERT-base model as the encoder. We use learn- ing rate of 3c-5 and apply a lincar warm-up and learning rate schedule with the weight decay value 0.01. The batch size is 4. We train the baseline models on 4 NVIDIA RTX A6000 GPUs for 10 epochs.
Fengqi Wang, Fei Li, Hao Fei. Jingye Li, Shengqiong Wu, Fangfang Su, Wenxuan Shi, Donghong Ji, and Bo Cai. 2022. Entity-centered cross-document re- lation extraction. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 9871 9871-9881, Abu Dhabi, United Arab Emirates. Association for Computational Lin-
A.2 Implementation Details
guistics.
To select text snippet from the reasoning path, follow the previous method (Yao et al., 2021). which extracts text snippets from the first mention of head and tail entities in each document. A slight difference from the previous method lies in our use of the bridging entity's 's document. In order to extract additional text snippets from the bridging entity's 's document as well, we augment the origi- nal text snippets by incorporating sections where the head and tail mentions first appear from the bridging entity's document.
Haoran Wu, Xiuyi Chen, Zefa Hu, Jing Shi, Shuang Xu, and Bo Xu. 2023. Local-to-global causal reasoning for cross-document relation extraction. JEEE/CAA Journal of Automatica Sinica, 10(7):1608-163 1621.
Hang Yang, Dianbo Sui, Yubo Chen, Kang Liu, Jun Zhao, and Taifeng Wang. 2021. Document-levs level event extraction via parallel prediction networks. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing [Volume| 1: Long Papers), pages 6298- 6308, Online. Association for Computational Lin- guistics.
A.3 Evaluation Metries
As evaluation metrics, we utilize F1/AUC/P@ 500/P@ 1000 for the develop- ment set and FI/AUC for the test set following the previous studies (Yao et al., 2021; Wang et al., 2022). All of the test results are obtained from the official website of CodRED on Codalab.
Yuan Yao. Jiaju Du, Yankai Lin, Peng Li, Zhiyuan Liu, Jie Zhou, and Maosong Sun. 2021. Codred: A cross- document relation extraction dataset for acquiring knowledge in the wild. In Proceedings of the 2021 Conference on Empirical Methods Natural Lan- guage Processing, pages 4452-4472.

---

A.4 Dataset Statisties
Train
Dev
Positive facts
2,733
1,010
1,012
Task Instruction
N/A 16,668
4,558
4,523
Bridges
613,566
195,766
197,888
Between the two the case where 1) the 'Text snippet' is more informative for inferring the 'Relation' and the where the 'Relation' and the 'Text snippet'
and B, choose
reasoning paths
129,548
40,740
40,524
Table 5: Statistics of CodRED.
relevant.
Examples
Train
Train (dist.)
Dev
Test
Example 1:
query-doc pairs
9,042
95,310
4.601
4,576
Case A:
Subject: Europa Plus
Table 6: Statistics of the Entity- aware Retriever dataset.
Object: Soviet Union
Relation: Europa Plus is country of Soviet Union.
Text snippet: The most titled volleyball tean in the Soviet Union and in Europe (CEV
# has bridge
bridge n/a posilive
n/a
posilive
Champions League) is VC CSKA Mascow.
40221
519
Case B:
2553
37668
524
Subject: Past Masters
Object: rock music
Table Statisties bridging entities for constructed path using PILOT.
Relation: Past Masters is of rock music.
of thirteen Beatles studio albums, Past Masters, and the "Red" and "Blue" greatest-hits albums made available Feelgood have been highly positive. Critics remarked the renewed energy album, bringing
Text snippet: 2010, the official
B Ch Evaluation Prompt Template
iTunes. Reviews for Dr.
entertaining
values permeate listeners "in world of everlasting party", where they "savored the joys trashy, unapologetically decadent fun.
1) Informativeness: A
2) Relevancy:
###
*##
Input
Case A:
Relation: (subject_entity.. is (relation_a)
of {object_entity_a))
Text snippet: treas))ing_text_A)
Case B:
Relation: (subject_entity_b) is (relation_b) of [object_entity__]]
Text snippet: {reasoning_texs text__
1) Informativeness:
2) Relevancy:
Table 8: Prompt Template for ChatGPT evaluation. We
provide three gold examples few-shot manner.


Reviews:
[{"review_id": "aBiOaACc3S", "reviewer": "Reviewer_shPs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments and suggestions. We sincerely appreciate your time reading the paper, and our responses to your comments are below.\n\n&nbsp;\n\n    Reasons To Reject: The performance of PILOT is deeply dependent on the quality of the initial reasoning path.\n\nRe: Because CodRED dataset has only 6.7% of positive reasoning paths and the remaining 93.3% of n/a reasoning paths, it can be said that PILOT can be employed to identify informative reasoning paths out of noises.\n\nBased on your opinion, we performed additional analysis to validate the efficacy of our method by investigating our results focused on the extremely noisy entity pairs that have more than 5 n/a paths with only 1~2 positive paths from the development set..\n\n|Method|Path-level accuracy|\n|---|---|\n|End-to-End|79.24|\n|+ PILOT|**80.31**|\n\nWe observed that our method obtains the higher path-level accuracy, which implies the higher capability to identify an informative reasoning path despite the noise.\n\nLastly, we agree that open-setting of CodRED, which requires constructing initial reasoning paths from scratch and also needing to reason the relations, is an important research field. However, this topic falls outside the scope of this paper, and we will leave this content for further researchers to explore.\n\n&nbsp;\n\n    Reasons To Reject: The proposed mothed neglect potentially important context when calculating the function score using the number of mentions in each document.\n\nRe: our main idea is about modeling how people search for information in the web to the process of reasoning path construction. When we search, we start with a query that represents what we want to find, and then we look at results based on the entity related to intention. This is similar to how PILOT works. So, in our approach, it's important to see the scoring method as part of a framework that combines with the dense retrieval. We are committed to providing a more comprehensive explanation of our approach in the revised manuscript.\n\nIn addition, it's worth noting that several studies have demonstrated the effectiveness of combining the dense representation with the sparse representation [1][2].\n\n\n&nbsp;\n\nReferences\n\n[1] Glass, M., Rossiello, G., Chowdhury, M. F. M., Naik, A., Cai, P., & Gliozzo, A. (2022, July). Re2G: Retrieve, Rerank, Generate. *In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*.\n\n[2] Dhingra, B., Zaheer, M., Balachandran, V., Neubig, G., Salakhutdinov, R., & Cohen, W. W. (2019, September). Differentiable Reasoning over a Virtual Knowledge Base. *In International Conference on Learning Representations.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Present an effective way of constructing explicit and structural reasoning path utilizing bridging entities for effective cross-document relation extraction.\n2. Provide an analysis with a large language model including ChatGPT on whether PILOT leads to more knowledgeable and relevant information.\n3. Achieved consistent improvement in performance compared to the baselines in the CodRED task.", "reasons_to_reject": "1. The performance of PILOT is deeply dependent on the quality of the initial reasoning path.\n2. The proposed mothed neglect potentially important context when calculating the function score using the number of mentions in each document.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper the authors introduce an effective way of constructing explicit and structural reasoning path utilizing bridging entities, named PILOT. PILOT first finds the bridging entities with structural entity information, and retrieves the documents that are related to the bridging entities. Afterward, it reranks the documents by entity-aware scoring function and leverage the high-scored documents as the stepstones connecting the head and tail. PILOT achieved consistent improvement in performance compared to the baselines in the CodRED task. In addition, the authors provide an analysis with a large language model including ChatGPT on whether PILOT leads to more knowledgeable and relevant information.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VXW2lxbbaU", "reviewer": "Reviewer_18D4", "summary": "", "questions": "A: Concerning the ChatGPT evaluation in Section 3.5: Did you conduct experiments to ensure the model's assessment of Informativeness and Relevancy matches your definition? Could you provide a baseline comparison between the “shared” and “all” approaches, or against random text?\n\nB: You mention using a dense retriever trained on Wikipedia articles, which retrieves documents related to a specific bridge entity, but discards those not directly mentioning the entity. Further documents are discarded if the wikipedia articles does not indicate a connection. One would expect that such strict filtering negates the benefits of a dense retriever. The description in the paper implies that entity mentions must exactly match the bridge entity. Could you please clarify how this works and whether entity disambiguation is used to identify instances that an exact match approach might miss?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply thank the reviewer for their _insightful comment_ and detailed feedback on _evaluation_, _typos_, and _presentation improvements_. They are exceedingly helpful for us to improve our paper. Please find our responses in the following.\n\n&nbsp;\n\n    Reasons To Reject: Modeling ambiguity The authors conflate the concepts of entity and an entity's wikipedia/wikidata text, which is questionable from a modeling perspective. As a result, it is also unclear how generalizable the proposed method is to open world relation extraction. While the concept of finding intermediate entities to identify further relaion paths between entities is interesting, it is not obvious to which degree this is relevant in practice. For example, consider Figure 1: while entities dh and dt are related, this relation is evident from the text of db - this does not mean that db has to be a linking entity, though, as it could be an arbitrary non-entity text. At the very least, the authors should clean up the conceptual approach an make clear how much of their model is specific to the wikidata/wikipedia ecosystem.\n\n\nRe: To clarify our method, we first note that our main idea is to reflect human's search process into the reasoning path and the importance of utilizing entity-centric information such as hyperlinks (Linking Entity) to identify helpful bridging entities in the cross-document. As you mentioned, in the real world, there are many non-entity texts along with erroneous (also misaligned) entities which could be helpful to infer the relation.\n\nHowever, our study begins with the assumption that effective reasoning paths can be constructed by following explicit and structural information, such as hyperlinks, much like the process humans employ during their searches. To put it differently, we drew inspiration from the fact that people often reach their objectives more efficiently by utilizing information like hyperlinks rather than relying solely on a single search.\n\nDefinitely, considering this kind of entity without structural information could indeed contribute to enhancing the quality of reasoning paths.\nHowever, this specific perspective wasn't the primary focus of our current study. We'd like to clarify that it's something we could explore in the future, especially in the more \"in the wild\" environments you suggested. We greatly appreciate your input on this idea.\n\nFurthermore, research involving entity-link information isn't limited to the domains of Wikipedia/Wikidata. By integrating existing systems like entity-linking and Named Entity Recognition systems into our method or developing a framework, we believe our approach could be extended to numerous other domains as well.\n\nIn conclusion, while our study is not focused on addressing situations lacking structured information like hyperlinks, we hope you'll consider our motivation to effectively construct reasoning paths by leaveraging such explicit and structural information.\n\n&nbsp;\n\n    Reasons To Reject: Evaluation The authors repeteadly mention the significance of their results, with no evidence of statistical testing. Some results (e.g. Figure 4) are very close and require significance testing to support the author's claims.\n\n\nRe: We sincerely appreciate the constructive feedback. We acknowledge the importance of statistical validation to support our claims, particularly in Figure 4 (Section 3.5). Based on your suggestions, we have taken steps to include statistical testing results in that section to bolster our claims.\n\nFor the experimental setting, we performed 7 different random samples and analyzed them with different random seeds.\n\n|                 |      Informativeness     |    |                           |   | Relevancy |    |                           |\n|:---------------:|:------------------------:|:--:|:-------------------------:|---|:---------:|:--:|:-------------------------:|\n|    **Method**   |         Win rate         |  | Confidence Interval (95%) |   |  Win rate |  | Confidence Interval (95%) |\n| Ours vs. Shared |          83.18%          |    |      81.88% ~ 84.48%      |   |   81.56%  |    |      80.22% ~ 84.48%      |\n|   Ours vs. All  |          82.66%          |    |      81.37% ~ 83.95%      |   |   81.45%  |    |      80.12% ~ 82.78%      |\n|  Shared vs. All | 79.85% (shared win rate) |    |      78.42% ~ 81.29%      |   |   76.22%  |    |      74.70% ~ 77.74%      |\n\n\n\nIn addition, we will revise our paper to include all of statistical validation in the final version. For example, we provide additional statistical results into the Table 1.\n\n| **Method**     |                        | **Dev**        |                |                |                | **Test**       |                |\n|----------------|------------------------|----------------|----------------|----------------|----------------|----------------|----------------|\n|                |                        | **AUC**        | **F1**         | **P@500**      | **P@1000**     | **AUC**        | **F1**         |\n| **End-to-End** |                        | 47.94          | 51.26          | 62.80          | 51.00          | 47.46          | 51.02          |\n|                | **+PILOT**             | 53.23 (± 0.59) | 56.12 (± 0.7)  | 70.86 (± 0.92) | 55.57 (± 0.25) | 54.31 (± 1.23) | 57.33 (± 1.31) |\n|                | **+PILOT (-Wikidata)** | 52.98 (± 0.98) | 55.72 (± 0.24) | 72.72 (± 0.8)  | 55.60 (± 0.21) | 53.37 (± 0.53) | 57.52 (± 1.01) |\n\n\n&nbsp;\n\n    Reasons To Reject: \"Manual\" LLM evaluation The authors use ChatGPT prompting instead of a human error analysis, which seems questionable in its efficacy. In particular, the authors note that ChatGPT favoured the extracted relationship description produced by this paper's method over the baseline by approximately 65%. This outcome was determined by presenting ChatGPT with two relationship descriptions and some task examples, asking the model to choose the more \"informative\" and \"relevant\" option. However, as ChatGPT's performance in such evaluations is still under investigation and prompt design will heavily influence results, alternative evaluation methods should be used. While LLMs like ChaGPT might be useful in method design, they should not be expected to replace human evaluation quie yet.\n\nRe: To address this issue, we also conducted a human evaluation using 30 samples with 3 people, as shown in the table below:\n\n|            |     | Informativeness |      |     | Relevancy |      |\n|:----------:|:---:|:---------------:|:----:|:---:|:---------:|:----:|\n|            | Win |       Tie       | Lose | Win |    Tie    | Lose |\n| vs. shared |  55 |        28       |   7  | 53  | 30        | 7    |\n| vs. all    |  61 |        27       |   2  | 59  | 29        | 2    |\n\nAccording to our human evaluation results, participants expressed a preference for PILOT's reasoning path in terms of Informativeness (61.1% for vs. shared and 67.7% for vs. all) and Relevancy (58.9% for vs. shared and 65.5% for vs. all).\nWe observed that these tendencies align with the results in the ChatGPT evaluation.\n\nFurthermore, while there is ongoing debate about the reliability of automated evaluations using LLMs such as ChatGPT, many studies do assert their validity [1, 2]. Particularly, in our study, entrusting the assessment of Informativeness and Relevancy to ChatGPT was motivated by the inspiration that our approach remains within the scope of research aligned with these findings.\n\nAs you've advised, we certainly do not disregard the importance of human evaluation. We are well aware of its significance. Therefore, we are conducting evaluations on a larger scale for Human Evaluation, and we intend to incorporate these evaluation results into the final version of this work.\n\nFinally, we will ensure to provide comprehensive explanations in the final copy regarding the guidelines, results analysis, and evaluation environment for human evaluation. \n\n\n&nbsp;\n\n    Questions For The Authors A: Concerning the ChatGPT evaluation in Section 3.5: Did you conduct experiments to ensure the model's assessment of Informativeness and Relevancy matches your definition? Could you provide a baseline comparison between the “shared” and “all” approaches, or against random text?\n\nRe: We appreciate your question, which focuses on the quality of the ChatGPT evaluation in Section 3.5. As we mentioned in our previous response, we conducted a human evaluation that provides support for the assessment conducted through ChatGPT.\n\nIn addition, as you requested, we provide a baseline comparison between the \"shared\" and \"all\" approaches using same criteria.\n\n|                 |      Informativeness     |    |                           |   | Relevancy |    |                           |\n|:---------------:|:------------------------:|:--:|:-------------------------:|---|:---------:|:--:|:-------------------------:|\n|    **Method**   |         Win rate         |  | Confidence Interval (95%) |   |  Win rate |  | Confidence Interval (95%) |\n|  Shared vs. All | 79.85% (shared win rate) |    |      78.42% ~ 81.29%      |   |   76.22%  |    |      74.70% ~ 77.74%      |\n\nThis result shows that PILOT is preferred by approximately 2.8%p than Shared baseline in the setting of comparison with the All baseline.\n\n&nbsp;\n\n    Questions For The Authors B: You mention using a dense retriever trained on Wikipedia articles, which retrieves documents related to a specific bridge entity, but discards those not directly mentioning the entity. Further documents are discarded if the wikipedia articles does not indicate a connection. One would expect that such strict filtering negates the benefits of a dense retriever. The description in the paper implies that entity mentions must exactly match the bridge entity. Could you please clarify how this works and whether entity disambiguation is used to identify instances that an exact match approach might miss?\n\nRe: We appriciate your detailed question for our study. We used strict filtering at the entity level in our study because it aligns with how human search processes often begin with specific queries and then expand to consider entities or keyword-based units. This emphasizes the importance of initial searches followed by utilizing structural information like hyperlinks.\n\nJust like your concern, we also worried that strict filtering might cause information loss. In initial experiments, we attempted to use surface-level information like string matching for filtering instead of entity id. However, we found that this approach introduced more noise than the potentially useful information it could provide for constructing reasoning paths. The cost of building reasoning paths seemed to outweigh the benefits.\n\nAdditionally, the CodRED dataset utilizes Entity ID from Wikidata (e.g., Q985235 for Fridley) for entity linking [3]. They employed a BERT-Large model trained on DocRED for Named Entity Recognition. Then, they aligned the recognized entity mentions with Wikidata's metadata like names and aliases for linking purposes.\n\nIn summary, our focus was on capturing the efficiency of constructing reasoning paths by combining dense retrieval with structural information like hyperlinks, akin to how humans often proceed with searches. While non-link-related entity information does exist, we hope our motivation behind the synergy of dense retrieval and structural information, as well as our key results, shines through in this context.\n\n\n&nbsp;\n\nReferences\n\n[1] Chiang, C. H., & Lee, H. Y. (2023). Can Large Language Models Be an Alternative to Human Evaluations?. *In Proceedings of the 2023 Conference on Association for Computational Linguistics*.\n\n[2] Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023). Gpteval: Nlg evaluation using gpt-4 with better human alignment. *arXiv preprint arXiv:2303.16634*.\n\n[3] Yao, Y., Du, J., Lin, Y., Li, P., Liu, Z., Zhou, J., & Sun, M. (2021, November). CodRED: A cross-document relation extraction dataset for acquiring knowledge in the wild. *In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Task relevance\nThe considered cross-domain relation extraction task is challenging and timely\n\n* Comprehensive model description\nThe path construction process is described in detail and the underlying reasoning is provided.\n\n* Evaluation against SOTA baselines\nPILOT is evaluated using two current state-of-the-art approaches, demonstrating an improvement in performance in both cases", "reasons_to_reject": "* Modeling ambiguity\nThe authors conflate the concepts of entity and an entity's wikipedia/wikidata text, which is questionable from a modeling perspective. As a result, it is also unclear how generalizable the proposed method is to open world relation extraction. While the concept of finding intermediate entities to identify further relaion paths between entities is interesting, it is not obvious to which degree this is relevant in practice. For example, consider Figure 1: while entities dh and dt are related, this relation is evident from the *text* of db - this does not mean that db has to be a linking entity, though, as it could be an arbitrary non-entity text. At the very least, the authors should clean up the conceptual approach an make clear how much of their model is specific to the wikidata/wikipedia ecosystem.\n\n* Evaluation\nThe authors repeteadly mention the significance of their results, with no evidence of statistical testing. Some results (e.g. Figure 4) are very close and require significance testing to support the author's claims.\n\n* \"Manual\" LLM evaluation\nThe authors use ChatGPT prompting instead of a human error analysis, which seems questionable in its efficacy. In particular, the authors note that ChatGPT favoured the extracted relationship description produced by this paper's method over the baseline by approximately 65%. This outcome was determined by presenting ChatGPT with two relationship descriptions and some task examples, asking the model to choose the more \"informative\" and \"relevant\" option. However, as ChatGPT's performance in such evaluations is still under investigation and prompt design will heavily influence results, alternative evaluation methods should be used. While LLMs like ChaGPT might be useful in method design, they should not be expected to replace human evaluation quie yet.", "questions_to_authors": "A: Concerning the ChatGPT evaluation in Section 3.5: Did you conduct experiments to ensure the model's assessment of Informativeness and Relevancy matches your definition? Could you provide a baseline comparison between the “shared” and “all” approaches, or against random text?\n\nB: You mention using a dense retriever trained on Wikipedia articles, which retrieves documents related to a specific bridge entity, but discards those not directly mentioning the entity. Further documents are discarded if the wikipedia articles does not indicate a connection. One would expect that such strict filtering negates the benefits of a dense retriever. The description in the paper implies that entity mentions must exactly match the bridge entity. Could you please clarify how this works and whether entity disambiguation is used to identify instances that an exact match approach might miss?", "paper_topic_and_main_contributions": "The authors introduce an enhanced method for cross-document relation extraction, termed PILOT. This technique aims to discern relationships between two entities across multiple documents. Unlike traditional methods, PILOT does not solely rely on information from documents where the head (h) and tail (t) entities appear, but instead utilizes so called 'bridging entities' that frequently cooccur with head and tail to form an extended reasoning path across multiple documents. Relevant documents containing these bridging entities are sourced using a dense retriever, trained on Wikipedia data with contrastive learning. The retrieved documents are then re-ranked to pinpoint those most relevant for discerning the relationship between head and tail entities. Documents lacking the bridge entity are excluded, as are bridging entities without Wikipedia structural information linking it to both entities. After re-ranking, the best bridge entity candidates are used to expand the existing reasoning path. Experiments demonstrae that existing approaches combined with PILOT achieve better performance as measured by the CodRED benchmark dataset.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "-"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The paper is overall well written, but contains numerous typos that should be corrected\n* line 035: Remove the extra “the.”\n* line 113-115: The sentence regarding how CodRED and Wikipedia were used to create the dataset for training the dense retriever is rather unclear.\n* line 123: It seems a “-” superscript might be missing from p_j on the equation's left side.\n* line 147: Equation 2 seems redundant, given the prior description about \"counting the number of occurrences.\"\n* Figure 3: Consider rephrasing “Achievement results on F1” to “Performance measured by F1 score for each type of …”\n\nFurther remarks:\n* The paper would benefit from a more detailed description of the key characteristics of the dataset \"CoDRED\".\n* Section 3.5 would benefit from a description of the extraction process for text snippets from the identified documents\n* In Table 1, why are only precision values only given for P@500 and P@1000 and not much smaller values?"}}, {"review_id": "GJ29vhUJH2", "reviewer": "Reviewer_F5va", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments. We sincerely appreciate your time in reading the paper. Please find the responses below.\n\n&nbsp;\n\n    Reasons To Reject: More baselines are needed.\n\nRe: We genuinely appreciate your valuable suggestion. To the best of our knowledge, regarding CodRED, we have employed all the available baselines (End-to-End[1] and ECRIM[2]) for our evaluations. Given that CodRED itself is not a task that has been in existence for an extended period, the availability of additional potential alternatives is limited.\n\n\n&nbsp;\n\n    Reasons To Reject: The presentation of the CodRED dataset is missing.\n\nRe: We value your feedback concerning the presentation of the CodRED dataset. We aimed to convey the fundamental schema of the reasoning path in CodRED through Figure 1. Moreover, we have included dataset statistics in the Appendix. We apologize if the explanation for enhancing the understanding of the dataset appeared insufficient.\n\nWe will provide detailed explanations of the CodRED dataset in the final version.\n\n&nbsp;\n\n    Reasons To Reject: I recommend verifying the effectiveness of PILOT on the more datasets.\n\nRe: We greatly appreciate your review and suggestion. Given that the CodRED task is designed to simulate a unique scenario of cross-document relation extraction using the reasoning path ceocept, there are no supplementary datasets available for method validation.\n\nNevertheless, as you have rightly pointed out, we will certainly consider exploring the scalability of our approach to other multi-document tasks as part of our future research endeavors.\n\n&nbsp;\n\nReferences\n\n[1] Yao, Y., Du, J., Lin, Y., Li, P., Liu, Z., Zhou, J., & Sun, M. (2021, November). CodRED: A cross-document relation extraction dataset for acquiring knowledge in the wild. *In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*.\n\n[2] Wang, F., Li, F., Fei, H., Li, J., Wu, S., Su, F., ... & Cai, B. (2022, December). Entity-centered Cross-document Relation Extraction. *In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper proposes a coherent story.\n2. The RE models with PILOT show performance improvement in comparison to the baselines within the CodRED task.", "reasons_to_reject": "1. More baselines are needed.\n2. The presentation of the CodRED dataset is missing. \n3. I recommend verifying the effectiveness of PILOT on the more datasets.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The cross-document relation extraction task involves deducing relationships between head and tail entities across various documents. Unfortunately, prior studies have overlooked the utilization of hyperlinks or supplementary searches to acquire relations of entity pairs. Therefore, this paper propose PILOT that exploring the explicit clue information within the documents by utilizing bridging entities. Experiments on three datasets verify its effectiveness.\n\n\n\n\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviews have a consensus on slightly positive soundness and excitement.

Regarding soundness, the empirical observations show the proposed method improves two state-of-the-art baselines in the CodRED task, demonstrating the effectiveness of the proposed method.  The technical description on the path construction process is comprehensive and clear.

With respect to excitement, the idea of employing bridging entities for reasoning paths is novel.  However, its practical impact is obscure.
Decision: Accept-Findings