Semi-automatic Data Enhancement for Document-Level Relation Extraction with Distant Supervision from Large Language Models
Authors: Junpeng Li
Zixia Jia
Zilong Zheng
Venue: EMNLP 2023 Main (2023)
Keywords: Long Context，Document-level，Relation Extraction
Large Language Model

Abstract: Document-level Relation Extraction (DocRE), which aims to extract relations from a long context, is a critical challenge in achieving fine-grained structural comprehension and generating interpretable document representations.
Inspired by recent advances in in-context learning capabilities emergent from large language models (LLMs), such as ChatGPT, we aim to design an automated annotation method for DocRE with minimum human effort.
Unfortunately, vanilla in-context learning is infeasible for DocRE due to the plenty of predefined fine-grained relation types and the uncontrolled generations of LLMs. To tackle this issue, we propose a method integrating an LLM and a natural language inference (NLI) module to generate relation triples, thereby augmenting document-level relation datasets.
We demonstrate the effectiveness of our approach by introducing an enhanced dataset known as DocGNRE, which excels in re-annotating numerous long-tail relation types. We are confident that our method holds the potential for broader applications in domain-specific relation type definitions and offers tangible benefits in advancing generalized language semantic comprehension.

Semi-automatic Data Enhancement for Document-Level Relation Extraction with Distant Supervision from Large Language Models
Junpeng Li', Zixia Jia", Zilong Zheng
National Key Laboratory of General Artificial Intelligence, BIGAI {lijunpeng, jiazixia,aaa.ai..i 2lah https: //gi thub. wwtp://aaiaiinnnoooooooo/aa
Abstract
DocGNRE (Ouns
2500
Re-DocRED
Document-level Relation Extraction (DocRE), which aims to extract relations from long context, is a critical challenge in achieving fine- grained structural comprehension and gener- ating interpretable document representations. Inspired by recent advances in in-context learn- ing capabilities emergent from large language models (LLMs), such as ChatGPT, aim to design an automated annotation method for DocRE with minimum human effort. Unfortu- nately, vanilla in-context learning is infeasible for document-level Relation Extraction (RE) due the plenty of predefined fine-grained re- lation types and the uncontrolled generations LLMs. To tackle this issue, we propose method integrating Large Language Model (L.M) and a natural language inference (NLI) module to generate relation triples, thereby aug- menting document- demonstrate the effectiveness of our approach by introducing an enhanced dataset known as DocGNRE, which excels in re-annotating nu- merous long-tail relation types. We are confi- dent that method holds the potential for broader applications in domain Nomain-specing specific rela- tion type definitions and offers tangible bene- fits in advancing generalized language semantic comprehension.
DocRED
2000
Count
1500
1000
P17 P27 P150P276P170PP P749 P551 P1366
Relation
Figure Counts of relation types in different datasets.
The original document-level RE dataset Do- cRED (Yao et al., 2019) has been recognized for its false negative issue and subsequently re-annotated to address this concern by supplementing sig- nificant number of relation triples. Notably, two representative works, Huang al. (2022) and Tan al. (2022b), have contributed this re-annotation process. Huang et (2022) undertook manual an- notation from scratch, employing two expert an- notators to annotate 96 documents. On the other hand, Tan et al. (2022b) utilized pre-trained RE models in conjunction with manual revision to con- struct Re-DocRED Despite their contributions to supplementing relations for DocRED, both meth- ods have certain limitations. First, achieving com- plete manual annotation is challenging: each doe- ument within this dataset contains an average 19.5 entities, requiring consideration of approxi- mately 37,000 candidate triples (including 97 re- lation types, including NULL). Second, the supple- mentary annotations are derived from the existing data distribution: Tan et al. (2022b) first pre-trained RE model with distantly supervised data from DocRED, then utilized this model to predict triple candidates. Such a process may introduce model bias, potentially resulting in the exclusion of sparse relations that exist beyond the scope of the existing
relation datasets. We
1 Introduction
Document-level Relation Extraction (DocRE) is task that focuses on extracting fine-grained rela- tions between entity pairs within lengthy context (Yao et al., 2019; Nan al., 2020; Wang al., 2020; Zhou al., 2021; Zhang al., 2021; Ma al., 2023). The abundance of entity pairs in document, coupled with a vast array of fine-graine graingg relation types, makes DocRE inherently more chal- lenging than sentence-level RE. The challenge is observed not only in model learning but also in human annotations.
Equal contributions. Authors ordered alphabetically. g Correspondence to Zilong Zheng <zizheng @ @iigui.ai>.

---

data distribution. Figure 1 illustrates the counts of some relation types across various datasets. It is evident that the supplementary annotations in Re- DocRED exhibit a distribution similar to that of the DocRED test set.
Our contributions can be summarized follows: We conduct a quantitative analysis evaluate the performance of GPT in zero-shot document- level RE.
D
We propose a novel framework that integrates an NLI module and an LLM to automatically generate distant relation triples, supplementing existing DocRE datasets.
In light of the limitations inherent in prior an- notation techniques and the imperative to enhance the completeness of DocRE datasets, we propose a novel approach aimed augmenting the Re- DocRED dataset through the utilization of the pow- erful generalization capabilities of Large Language Models (LLMs). By leveraging our method, as shown in Figure 1, our revised test set, named DocGNRE, exhibits the advantage of re-annotating greater number of long-tail types, such as P276 and P551.
We create an enhanced version of the Re- DocRED test set, named DocGNRE, with mini- mal human intervention, ensuring high quality. Additionally, we augment the Re-DocRED train- ing set by supplementing it with distant relation triples automatically generated by our frame- work! referring to Tabel 1.
LLM Enhanced Automatic Data Generation
Recent studies have utilized GPT (Floridi and Chiriatti, 2020; Chan, 2023) for various structural prediction tasks, such as named entity prediction and relation extraction (Dunn et al., 2022; Gutiér- rez et al., 2022; Wang et al., 2023; Liu et al., 2023; Xu et al., 2023), well text classification label- ing (Gilardi al., 2023; Törnberg, 2023). Notably, researchers such as Wan al. (2023); Wadhwa al. (2023) have demonstrated the effectiveness of in-context learning by incorporating prompts containing suitable example demonstrations for RE tasks. However, it is worth noting that without ex- plicit instructions, GPT may generate uncontrolled relations that do not align with predefined types. Therefore, recent methods only work in sentence- level RE and especially highlight one distinguished challenge for LLM in-context learning: unable to fitting detailed instructions for long-context docu- ments (Wadhwa et al., 2023).
Our approach consists of two main procedures: constructing prompts for LLM to generate rela- tions triples as proposals and employing Natural Language Inference (NLI) models to align gener- ated relations with predefined relation types. Fig. 2 shows the whole framework. In the first procedure, observed that even though we imposed restric- tions on the entity and relation lists in the prompts, LLMs (both GPT-3.5 and GPT-4) still generated triples that fell outside of our intended constraints. Furthermore, we found that the generated relations expressed by contextual words in the document were more accurate than those with the restricted relation types. Based on these insights and to fully leverage the potential of LLMs, hereby generat- ing more accurate relation triples, we removed the restrictions of specific relation types for LLMs. In- stead, we subsequently utilize an NLI module map the generated relations to the predefined rela- tion types in the Re- Re-DocREDD dataset.
To align GPT-generated relations and predefined relation types, we first combine Natural Language Inference (NLI) models (MacCartney, 2009) with GPT solve zero-shot DocRE. The results show that although GPT generations only hit partial ground truth of Re-DocRED, detects substantial external valid relation triples (details in Sec. 3.1). Therefore, we design a pipeline framework to fur- ther complement the test set of Re-DocRED and automatically generate distant training annotations by combining GPT and NLI modules. To verify that we supplement many relation triples beyond the scope of the original data distribution, we test previous models our DocGNRE test set. Addi- tionally, we train the statc-of-ther ahe (SOTA) model using our distant training dataset.
2.1 GPT Results as Proposals
We select GPT-3.5 (gpt-3.55 5-ttrbbb)) as our LLM module, considering a balance between cost and performance. Given that the original DocRED dataset provides an entity list for each document, we constrain the responses of GPT to utilize only the entities present in the provided list.
Prompt Construction As shown in Figure 2, the prompt consists of generation demonstration and a specific context followed by a corresponding
'Our dataset is publicly available at https: //gi thub. com/bigai maaaainncce -allimnccoooooo coDDGGR.,

---

Prompt Construction
GPT-generated relation triples
for', Films's
Please generate at least 20 triples that consi dered be true respect to below concontext using only given entities from the entity list. Please onswer using triples form of <' c'entityl', "relation', "entity2'> 'entityl' and 'entity2" below entity List.
Londonn Films', "wos owned by', 'Alexonder Korda'> The Sound Barrier', 'is at, 'British file's
worked Premiset B: "London Films was owned by Wexander Korda"
NLI Module
1. "The person or organization for which sub. [David
Context: The Sound Barrier known in the United States os Breaking Through Sound Barrier Breaking the Sound Barrier is 1952 British film directed Dovid Lean.
works or worked obj. (Lonzon ...!" <- Emplayer
Hypot hesis:
sub, ... works worked ob), (David ...! Emplayur
person or organization
3. "The owner of sub. is oij "
Owned by
it was his first for Alexander Korda 's London Films following the break
"sub. has obj, their ebbing"
Bsottng
<David ... Employer; London ...>
And Entity list: [8reakinng Through the Sound Barrier, 1952, David Lean, Alexander Korda, London Filns, ...]
<London.... Owneed by. David ... 0.22
<London.... Employer.
<David ... Sbiing, London ...> 0.01
Filtering
David Lean', Employer' London Films'
London Filns',
"Owned by". exaxnder Korda"
For example, <aCaribbnnn', 'part of', 'Atlantic'z, <'Aclantic', "has part', 'Caribbean's
<London Owned by, Alexander... 0.85
<lexsnderr... Employer,
<Nexander ... Sibling, London ...h 0.01
Figure 2:
The automatic data generation framework and exemplar document. The green triple in GPT-generated relation triples will be filtered because its object entity is out of the given entity list.
entity list. We notice that as the generated content by LLMs became longer, the accuracy decreased. To mitigate this, we set "at least 20 triples" in the initial prompt 2. To generate more additional triples, we employ an iterative approach by feeding the previous GPT answers as input while instructing GPT to "Please keep generating 20 more triples using only the given entities from the entity list". However, despite providing the entity list in the prompt, we observed that undesired triples with incorrect entity pairs still occurred. To address this, we implemented a filtering process to remove these undesired triples. Consequently, all the remaining triples are treated as proposals and later aligned using the NLI module.
cating whether it signifies entailment, neutrality, or contradiction. If the term "entailment" receives the highest score, the model concludes that the two sentences are factually consistent.
Premise and Hypothesis Construction our framework, we take each GPT-generated triple as the premise and replace the relation in such triple with a specific predefined relation type as the hy- pothesis. Remember that our purpose is map each GPT-generated relation proposal to prede- fined relation type. Hence, we should enumerate the hypothesis constructed by each specific type in the predefined set calculate the entailment scores with the corresponding premise and choose the ONE with the highest score. Moreover, we observe that the GPT-generated relation may cor- respond to an inverse predefined relation type. For example, if the predefined relation set contains the "employee" type rather than "employer", the GPT- generated triple "<David Lean, worked for, London Films>" will correspond to "<London Films, em- ployee, David Lean>" rather than "<David Lean, employee, London Films>". Therefore, for each generated relation proposal as a premise, we con- struet 96*2 = 192 possible hypotheses, where 96 is the size of the predefined relation set without NULL, and double means change the subject and object entities for each predefined type. Specifically, take triple < e1, e2 > generated from GPT an example premise, given the predefined relation set {r1, r..... we construct candidate hypotheses {< 61, F1, 62 >, < 622 T1, e >, C,,72, 62 >,
2.2 NLI Module
Annotator
this procedure, our goal is to map the relations generated by GPT to predefined types. To achieve this, reasonable approach is to align the seman- tic meaning of relations. Therefore, we employ NLI model, which has demonstrated effectiveness in assessing factual consistency (Honovich al., 2022). The NLI model takes two sentences as in- put, typically referred as the premise and the hypothesis. assigns a score to each term, indi-
"We choose the "20" number because it a trade-off be- tween ensuring accuracy and the quantity of generated re- lations. We find that the first 20 or so relations generated by GPT-3.55 (gpt-3.5- 5utubbo exhibit a relatively promising level of quality. The "at least 20" could be replaced by "no more than" and "a maximum of". We discovered that GPT-3.5 gen- crates comparable numbers and quality of triples using all of these expressions.

---

# Doc
# Ent
Through above procedures, we process each doc- ument of the Re-DocRED train set produce addi- tional distant relation triples. For the Re-DocRED test set, after acquiring distant relation triples, each 2,078
# Tri
Re-DocRED
500
9.779
17,448
Test
DocGNRE (Ours)
500
9,779
19,526
A
0
0
Re-DocRED
3,053
59,359
85,932
distant triple will be conducted through human verification. Two annotators are asked to answer whether the relation triples can be inferred accord- ing to the provided documents. A third annotator will resolve the conflicting annotations. Specifi- cally, we use Mechanical Turk for human annota- tions. In order to ensure that annotators possessed a significant level of qualification, prospective anno- tators were required to meet the following criteria: "HIT Approval (%) for all Requesters' HITs" 95.
Train
Re-DocRED+GIT
3,053
59,359
96,505
Re-DacRED+morc GPT
3,053
59,359
103,561
Table 1: Comparation of relation statistics between Re- Docred and DocGNRE, The test set has been verified by human annotators. The GPT GeTT-generated triples (+GPT) the train set distant.
e2, E1 >...}. Because the NLI model is pretrained with nat-
ural language sentences, we convert the triples to
natural sentences.
"Number of HITs Approved" > 1000. "Location" is one of (United States, Canada, Great Britain, Australia, Singapore, Ireland, New Zealand).
Most of the GPT-generated relations are them-
selves in natural language, each triple's 's subject entity, relation, and object entity are directly con- catenated to get natural sentence.
The first two indicators are calculated by Mechani- cal Turk according to one's historical performance and the last one aims to promise English profi- ciency of the annotators. Finally, the acceptance rate of the NLI-selected relations in the test set is 71.3%. We provide a more accurate and complete test set with the addition of 2078 triples than Re- DocRED, Detailed statistics of our datasets can be found in Table 1.
D> The predefined relation types typically are ab- stractive. To make the hypothesis precisely convey the meaning of each relation type, integrate the description of each relation type with subject and object entities. Hypothesis for each relation type can be found in Appendix C.
Entail Scores from NLI Model We use the T5-based NLI model in this paper for its powerful generalizability. T5-XXL (Raffel et al., 2020) is a generative model, which identifies "Entailment" and "No entailment" by generating two sequences in the inference stage. We leverage the probabilitics of such two sequences omitting the start and end tokens to calculate the entailment scores used for sorting predefined relation types. Details can be found in Appendix A.
3 Experiments
3.1 Zero-shot Document-level RE
Our framework can obviously be used to predict document-level relations directly. Therefore, in the first experiment, we explore the GPT performance on the zero-shot document-level RE. Table shows the results. As far as we know we are the first to report these results on document-level RE.
Post Processing To ensure the high quality of newly produced relation triples, we ultimately retain those hypothesis triples that should satisfy all the following principles:
We have three observations based on Table 2: i) Pure GPT-3.5 (without our NLI module) only hits rare ground truth. As aforementioned, GPT generates most relations expressed by natural lan- guage, which do not exactly match with the ground truth, even though some of these relations represent the same meaning as ground truth. So the exact- match Fl scores are unsatisfactory; ii) NLI module can improve pure GPT performance. With NLI module mapping GPT answers to predefined types, GPT-3.5 (gpt-3. 5-turbo) predicts small portion of ground truth triples that are manually annotated (5.77 recall in the Re-DocRED test set). The reason may be that we ask it to generate multiple relations
The entity types of subject and object entities satisfy the type constraints of the relation.
D Get the highest entailment scores.
D Get the entailment scores of more than 0.6. Note that some of the GPT- GPT-generated relations may be exactly those in the predefined set of rela- tion types. We do not need map these generated triples via our NLJ module and just add them into the final selected triples set.
hettps://huggnnfaass ace. co/googleett_xxx_.ero nii_mixture

---

DocRED
Re-DocRED
DocGNRE
Method
P
R
F1
P
F1
GPT-3.5 Only
7.34
4.53
5.6
13.12
2.85
4.68
13.97
2.71
4.54
GPT-3.5 + (W.. rel des)
13.9
10.29
11.82
23.57
6.14
9.74
42.91
9.9
16.2
GPT-3.5 NIJ (w. rel des)
14.61
9.8
11.73
24.45
5.77
9.33
72.71
15.32
25.31
Table 2: Results of zero-shot document-level RE. We test the three test sets: DocRED, Re-DocRED, and our DocGNRE. "rel des" means relation description. "P" and "R" refer to precision and recall respectively.
DocGNRE (D)
Re R
D Re
difficult prediction on our supplementary test rela- tion triples when the model is only trained with the training set of DocRED or ReDocRED; ii) The re- call scores on all the test sets are improved with
Train set
P
R
F1
R
DocRED f
90.3
27.83
42.55
31.04
DocRED $
84.52
32.1
46.52
35.73
1.54
Re-Dock Re-DocRED |
81.45
56.98
67.05
63.59
1.40
directly supervised training on our training set
Re- Re-DocRED t
85.0
64.29
73.21
71.69
(which exhibits the capability to predict additional ground truth instances), even though our distantly supervised data is somewhat noisy. Designing more advanced methods leverage our distant training is taken in future work.
Doc GPT
84.54
27.9
41.96
29.72
12,66
Doct GPT $
84.07
34.86
49.2
37.22
15.08
Doc mGPT
80.65
28.89
42.54
30.02
19.39
Doe mGPT
79.29
36.31
49.75
38.09
21.4
Re+ GPT
83,66
57.62
68.24
62,87
13.53
Re+ GPT
84.92
63.86
72.9
70.0
12.29
Re+ mGPT
81.71
58.23
68.0
62.74
20.36
In addition. we conducted experiments using two other DocRE models, ATLOP (Zhou et al., 2021)
Ret mGPT 80.93
66.98
73.29
72.36
21.86
Table 3: Results of DREEAM model (Ma et al., 2023) training with different settings. All results are averaged by three runs. "mGPT"" means "more GPT" that we carry out two iterative processes as mentioned in Sec, 2.1. "Doc" and "Re" are abbreviation of "DocRED" and "Re- DocRED". "D-Re" refers to our supplementary triple set (the remaining triples obtained by removing triples of Re-DocREDD from DocGNRE. means BERT-hase (Devlin et al., 2018) and $ means RoBERTa-large (Liu et al., 2019).
and KD-DocRE (Tan et al., 2022a), by leveraging their officially provided code. Experimental results of ATLOP and KD-DocRE show similar tendency to DREEAM. Detailed results are in Appendix B.
Conclusion
LLMs face challenges extracting fine-grained relations within lengthy contexts. To address this limitation, we present a novel framework that in- tegrates an NLI module in this work. With our framework, we improve the performance of GPT in zero-shot document-level RE. Above all, our framework enhances the automatic data generation capability with minimum human effort. We sup- plement the existing DocRE dataset, providing a complete test set DocGNRE and a distant training set. Given the inherent presence false negative in- stances in numerous RE datasets, particularly those scheme
at once by one prompt rather than enumerate all entity pairs to ask for relations one by one (which is too costly and time-consuming to execute for plenty of entities on document-level RE). But from human verification, the accuracy of NLI-selected triples has been proven relatively high (72.71 pre- cision in our supplementary test set DocGNRE), which illustrates that most triples predicted by our framework are the supplementary of Re- Re-DocRED; iii) Relation descriptions can guide NLI to out- put expected relations. With relation descriptions to construct hypothesises, the performance is fur- ther improved (25.31 31 vs. 16.2) in our DocGNRE.
constructed through a recommend- or distant supervision, we believe our framework possesses broad utility that extends to wider array of datasets.
3.2 Training with Distant Triples
Limitations
We test the SOTA document-level RE model (Ma et al., 2023) our distant training set. All experiment settings are the same as Ma et al. (2023) except for training data in the +GPT setting. Table 3 shows the results. We can find that i) the recall of previous models on our NRE drops, which demonstrates the
DocGNRE and retrain it with
The limited generated length of LLMs causes the limitation of our methods. There is specific upper limit on the number of relation triples that can be generated for each document. Therefore, our frame- work is an excellent data supplement method rather than a perfect zero-shot predictor.

---

Acknowledgements
Zhengliang Liu, Xiaowei Yu, Lu Zhang, Zihao Wu. Chao Cao, Haixing Dai, Lin Zhao, Wei Liu, Ding- gang Shen, Quanzheng Li, et al. 2023. Deid-gpt: Zero-shot medical text de-identification by gpt-4, arXiv preprint wrXiv2333..103:
This work is supported in part by the National Key R&D Program of China (2021ZD01 and the National Natural Science Foundation of China (62376031).
Youmi Ma, An Wang. and Naoaki Okazaki. 2023. Guiding attention with evidence for im- proving document-level relation extraction. In Pro- ceedings of the 17th Conference of the European Chapter of the Association for Computational Lin- guistics, pages 1971-1983, Dubrovnik, Croatia. As- sociation for Computational Linguistics.
DREEAM:
References
Anastasia Chan. 2023. Gpt-3 and instrucigpt: techno- logical dystopianism, utopianism, and "contextual" perspectives in ai ethics and industry. Al and Ethics,
3(1):53-64.
Bill MacCartney. 2009. Natural language inference. Stanford University.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint
Guoshun Nan, Zhijiang Guo, Ivan Sckulić, and Wei Lu. 2020. Reasoning with latent structure refinement for document-leved relation extraction. Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pages
Alexander Dunn, John Dagdelen, Nicholas Walker, Sanghoon Lee, Andrew S Rosen, Gerbrand Ceder, Kristin Persson, and Anubhav Jain. 2022. Struc- tured information extraction from complex scientific text with fine-tuned large language models. arXiv preprint 222::221...02238.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. The Journal of Machine Learning Reseurch, (1))54855-555.
Luciano Floridi and Massimo Chiriatti. 2020. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines, 30:681-694.
Qingyu Tan, Ruidan He, Lidong Bing, and Hwee Tou Ng. 2022a. Document-level relation extraction with adaptive focal loss and knowledge distillation. In Findings of the Association for Computational Lin- guistics: ACI. 2022, pages 1672-1681, Dublin, Ire- land. Association for Computational Linguistics.
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms crowd-workers for text- annotation tasks. arXiv preprint arXiv:2303.1 15056.
Bernal Jiménez Gutiérrez, Nikolas McNeal, Clay Wash- ington, You Chen, Lang Li, Huan Sun, and Yu Su. 2022. Thinking about gpt-3 in-context learning for biomedical ic? think again. arXiv preprint arXiv:2003.08410.
Qingyu Tan, Lu Xu, Lidong Bing, Hwee Tou Ng, and Sharifah Mahani Aljunied. 2022b. Revisiting docred addressing the false negative problem in relation extraction. In Annual Conference on Empirical Meth- ods in Natural Language Processing (EMNLP).
Or Honovich, Roee Aharoni, Jonathan Herzig. Hagai Taitelhaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim. and Yossi Matias. 2022. TRUE: Re Re-evaluating factual consistency evaluation. In Proceedings of the Second DialDoc Workshop on Document- seveeett-rroue grounded Dialogue and Conversational Question Answering, pages 161- 175, Dublin, Ireland. Association for Computational Linguistics.
Petter Tömberg. 2023. Chatgpt-4 outperforms experts and crowd workers in annotating political twitter messages with zero-shot learning. arXiv preprint arXiv:2304.06588.
Somin Wadhwa, Silvio Amir, and Byron C Wallace. 2023. Revisiting relation extraction in the era of large language models. arXiv preprint arX1v:2305.05003.
Quzhe Huang, Shibo Hao, Yuan Ye, Shengqi Zhu, Yansong Feng, and Dongyan Zhao. 2022. Does recommend-reviss produce reliable annotations? an analysis missing instances in docred. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2241-2252.
Zhen Wan, Fei Cheng, Zhuoyuan Mao, Qianying Liu, Haiyue Song, Jiwei Li, and Sadao Kurohashi. 2023. Gpt-re: In-context learning for relation extrac- tion using large language models. arXiv preprint urXiv:2305.021155
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.
Difeng Wang. Wei Hu, Ermei Cao, and Weijian Sun. 2020. Global-to-local neural networks for document- level relation extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 37 1-3721.
Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint trXiv:1907.1 16692.

---

Shuhe Wang, Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li, and Guoyin Wang. 2023. Gpt-ner: Named entity recognition via large language models. arXiv preprint arXiv:2304. 10428.
Xin Xu, Yuqi Zhu, Xiaohan Wang, and Ningyu Zhang. 2023. How to unleash the power of large lan- guage models for few-shot relation extraction? arXiv preprint arXiv:2305.011 01555.
Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019. DocRED: A large-scale document-leve relation extraction dataset. In Pro- ceedings of the 57th Annual Meeting of the Associa- rion for Computational Linguistics, pages 764-777, Florence, Italy, Association for Computational Lin- guistics.
Ningyu Zhang, Xiang Chen, Xin Xie, Shumin Deng, Chuanqi Tan, Mosha Chen, Fei Huang, Luo Si, and Huajun Chen. 2021. Document-lewe relation ex- traction as semantic segmentation. arXiv preprint arXiv:2106.0361..
Wenxuan Zhou, Kevin Huang. Tengyu Ma, and Jing Huang. 2021. Document-level relation extraction with adaptive thresholding and localized context pool- ing. In Proceedings of the AAAI conference artifi- cial intelligence, volume 35, pages 14612-1400 14620.

---

DocGNRE (D)
Re
D Re
B Model Results
Train set
F1
R
R
Experimental results of ATLOP and KD-DocRE are shown in Table 4 and Table respectively.
DocRED
87.9
43.2
31.9
0,9
DocRED 3
89.8
31.1
46.2
34.6
1.5
Doc + GPT
82.9
29.3
43.2
31.2
13.0
C Hypothesis Construction of Relation
Doc GPT
84.4
32.2
46.6
34.4
13.3
Doc + mGPT
79.6
30.2
43,8
31.5
19.54
We provide hypothesis construction of predefined relations in DocRED and Re-DocRED with Wiki- data Id and Name in Table
Doc + mGPT
79.3
34.2
47.7
35.7
21.1
Table 4: Results of ATLOP model (Zhou et al., 2021) training with different settings. i means BERT-base (De- vlin al., 2018) and means RoBERTa-large (Liu al., 2019).
DocGNRE (D)
Re
D
Re
Train set
F1
R
R
DocRED
82.4
32,9
47.0
36.6
1,6
DocRED
76.9
37.8
50.3
41.9
2.7
Doc + GPT
74.3
36.1
48.5
38.3
17.9
Doc GPT
77.1
37.4
50.4
39.7
18.8
Doc + mGPT
65.3
39.2
48.9
40.0
31.9
Doc + mGPT
68.8
51.6
42.5
30.8
Table 5: Results of KD-DocRE model (Tan al., 2022a)
training with different settings. it means BERT-base (De-
vlin al., 2018) and means RoBERTa-large (I.iu al., 2019).
A NLI Score
We choose the T5-based NLI model released by Google in this paper. As mentioned earlier, T5- XXL. is a generative model. The model identifies "Entailment" and "No entailment" by generating two sequences in the inference stage. During infer- ence, the sequence "<pad>_ 0a//>>00///>>/// identifies "No entailment", and the sequence "<pad>1 1 
<pad>> identifies "Entailment". Because both the first to- kens are "<pad>", and when the first three tokens have been determined, the prediction of the last token has a high probability, so we do consider the first and last token when calculating the NLI score. To obtain the NLI score, we obtain the logits of four subsequences ("_0", "=0",,"<<>>""" 
", "10", "1</>>") and perform softmax operation to obtain the corre- sponding probabilities. The score of "_0" sequence corresponds the score of "No entailment". The score of "1
" sequence corresponds to the score of "Entailment". To further distinguish the NLI score among the constructed triples, we subtract the score of "No entailment" from the score of "En- tailment" to fuse the two scores as the final scores.

---

Wikidata ID
Name
Hypothesis Construction
P6
head of government
The head of the executive power of the governmental body sub. is obj.
P17
country
The sovereign state this item sub. is obj.
P19
place of birth
The birth location of the person, animal or fictional character sub. is obj.
P20
place of death
The death location of the person, animal or fictional character sub. is obj.
P22
father
The father of sub. is obj.
P25
mother
The mother of sub. obj. The spouse of sub, is obj. obj. a country that recognizes sub. as its citizen
P26
spouse
P27
country of citizen-
ship
P30
continent
obj- is the continent of which sub. is a part
P31
instance of
obj. is that class of which sub. a particular example and member.
(sub. typically an individual member with proper name label)
P35
head of state
obj. is the official with the highest formal authority in the coun- try/state sub.
P36
capital
obj. is the primary city of the country/state sub. obj. is the language designated official by sub.
P37
official language
P39
position held child
sub. currently or formerly holds the position public office obj. sub. has obj. their offspring The main creator(s) of the written work sub. is(are) obj.
P40
daughter
P50
author
P54
member of sports team
The sports team or club that sub. represents or formerly repre- sented is obj.
P57
director
The director of this film, TV-series, stageplay video game is obj-
P58
screenwriter
The author(s) of the screenplay or script for this work sub. is(are)
obj.
P69
educated at composer
The educational institution attended by sub. is obj.
P86
The person(s) who wrote the music sub. is(are) obj.
P102
member of political party
The political party of which this politician sub. or has been a member is obj.
P108
employer
The person or organization for which sub. works or worked is obj.
P112
founded by
The founder co-founder of this organization, religion or place sub. is obj.
P118
league
The league in which the team or player sub. plays or has played in is obj.
P123
publisher
The organization or person responsible for publishing books, periodicals, games or software sub. is obj.
P127
owned by
The owner of sub. is obj.
P131
located in the ad- ministrative territo- rial entity genre
sub. is located on the territory of the following administrative entity obj.
P136
The creative work sub''s genre obj.
P137
operator
The person or organization that operates the equipment, facility, or service sub. obj.
P140
religion
The religion of person, organization religious building, associated with sub. is obj.
P150
contains administra- tive territorial entity
The direct subdivisions of an administrative territorial entity sub. has obj.

---

Wikidata ID
Name
Hypothesis Construction
P155
follows
The immediately prior item in some series of which sub, is part is obj.
P156
followed by
The immediately following item in some series which sub. is part is obj.
P159
headquarters loca- tion
The specific location where sub.'s 's headquarters is or has been situated is obj.
P161
cast member
The actor performing live sub. for camera or audience has obj. The producer(s) of this film music work sub. is(are) obj. The award or recognition received by person, organization creative work sub. obj.
P162
producer
P166
award received
P170
creator
The maker of a creative work sub. is obj. The closest parent taxon of the taxon sub. is obj. sub.'s 's ethnicity is obj.
P171
parent taxon ethnic group performer
P172
P175
The performer involved in the performance or the recoding of the work sub. is obj.
P176
manufacturer
The manufacturer producer of the product sub. is obj.
P178
developer
The organization person that developed sub. is obj. The series which contains sub. is obj.
P179
series
P190
sister city
sub. and obj. are twin towns, sister cities, twinned municipalities The legislative body governing sub. obj.
P194
legislative body
P205
basin country
The country that have drainage to/from or border the body of water sub. has obj.
P206
located in or next body of water military branch
sub. is located in or next to body of water obj.
P241
The branch to which the military unit, award, office, person sub. belongs is obj.
P264
record label
The brand and trademark associated with the marketing of subject music recordings and music videos sub. is obj.
P272
production pany location
com-
The company that produced this film, audio or performing arts work sub. is obj.
P276
The location of the item, physical object or event sub. is within is obj.
P279
subclass of subsidiary part of
All instances of sub. are instances of obj.
P355
The subsidiary of company or organization sub, has obj. obj. has part parts sub.
P361
P364
original language of work
The language which the film or a performance work sub. was originally created is obj.
P400
platform
The platform for which the work sub. has been developed or
released specific platform version fo the software sub. developed is obj.
P403
mouth of the water- course original network
The body of water to which the watercourse sub. drains is obj.
P449
The network(s) the radio or television show sub. was originally aired on has obj.
P463
member of chairperson
The organization or club to which sub. belongs obj.
P488
The presiding member of the organization, group or body sub. is obj.
P495
country of origin has part residence
The country of origin of the creative work sub. is obj.
P527
sub. has part or parts obj.
P551
The place where the person sub. is, or has been, resident is obj.
P569
date of birth
The date on which sub. was born is obj.
P570
date of death
The date on which sub. died is obj.

---

Wikidata ID
Name
Hypothesis Construction
PS71
inception
The date or point in time when the organizatioon/suject sub. was founded/created is obj.
P576
dissolved, abol-
The date or point in time on which the organization sub. was dissolved/disappearee or the building sub, demolished is obj. The data or point in time the work sub. is first published or released is obj.
ished or demolished publication date
P577
P580
start time
The time the item sub. begins to exist or the statement sub. starts being valid is obj.
P582
end time
The time the item sub. ceases to exist or the statement sub, stops being valid is obj.
P585
point in time
The time and date sub. took place, existed or the statement sub. was true obj.
P607
conflict
The battles, wars other military engagements in which the person item sub, participated is obj.
P674
characters
The characters which appear in sub. has obj.
P676
lyrics by
The author of song lyrics sub. is obj.
P706
located terrain
sub. is located on the specified landform obj.
feature participant
P710
The person, group of people organization that actively takes/took part in the event sub. has obj-
P737
influenced by location of forma- tion
The person, idea sub. is informed by obj.
P740
The location where the group organization sub. was formed is obj-
P749
parent organization
The parent organization of the organization sub. is obj.
P800
notable work
The notable scientific, artistic or literary work, or other work of significance among sub.'s 's works is obj.
P807
separated from
sub. was founded or started by separating from identified object obj.
P840
narrative location work location
The narrative of the work sub. is set in the location obj.
P937
The location where persons organization sub. were actively participating in employment, business or other work is obj.
P1001
applies to jurisdic- tion product or material produced unemployment rate
The institution, law public office sub. belongs to or has power applies the country, state or municipality obj.
P1056
The material or product produced by the government agency, business, industry, facility, or process sub. is obj.
P1198
The portion of the workforce population that not employed of sub. is obj.
P1336
territory claimed by
The administrative divisions that claim control of the given area sub. is obj.
P1344
participant of
The event that the person the organization sub. was partici- pant in is obj.
P1365
replaces replaced by capital of
The person or item sub. replaces obj.
P1366
The person or item obj. replaces sub. sub. is capital of obj.
P1376
P1412
languages spoken, written signed present in work
The language(s) that the person sub. speaks writes is obj.
P1441
The work in which the fictional entity or historical person sub. is present is obj. sub. has obj. as their sibling
P3373
sibling
Table Relation list, including Wikidata IDs, Names and Hypothesis Construction of relations


Reviews:
[{"review_id": "ykiGVEeyE0", "reviewer": "Reviewer_KqzM", "summary": "", "questions": "N/A", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thoughtful feedback and acknowledgment of our method.\n\n> In Table 2, Precision and F1 should also be shown for Re and D - Re datasets. What is 'D - Re' and 'Re'?\n\n**A:** As mentioned in the caption of Table 2, the term \"Re\" refers to the abbreviation of the \"Re-DocRED\" dataset; \"D-Re\" refers to our supplementary triple set (the remaining triples obtained by removing triples of Re-DocRED from DocGNRE).\n\nWe list the reasons for not reporting the precision and F1 of the Re-DocRED and the supplementary triple set as follows: \n- Annotations for **both \"Re\" and \"D-Re\" are partial** (\"Re\" and \"D-Re\" are two subcollections of the whole ground truth \"$D-Re \\cup Re = DocGNRE$\"), in which **some correct predictions are not counted as true positives**. Hence, separately calculating precision on these two sets offers limited insight. We choose to present all evaluation metrics for DocGNRE because it is more complete and the evaluation scores are more convincing. \n- We present the recall scores to demonstrate that the model, after retraining on our labeling distant training set, exhibits an improved capability to **predict additional ground truth instances**. This improvement is observed **not only within our generated test set but also across the previously annotated test set**.\n\n> Some previous work also annotates the dataset. The evaluation of other annotated datasets should be conducted.\n\n**A:** Please refer to the zero-shot GPT performance in **Table 1** on **DocRED** and **Re-DocRED** which are *annotated by previous work*, and different model performances on **Re-DocRED** test in **Table 2** by training with different datasets (DocRED, Re-DocRED and ours).\n\n> The quality of the annotated datasets should be shown. Some human evaluations should also be conducted. Some data statistics should be shown.\n\n**A:** As described in paper **Lines 234-239**, we conduct the human evaluation over our generated relation triples to construct the final test set in order to ensure the high quality of the dataset. More specifically, we use Mechanical Turk for human annotations. Notably, approximately **91.7% of the generated triples by GPT** and around **71.3% of NLI-selected predefined relations** were identified as positive instances.\"\n\nData statistics have been displayed in Table 3 in **Appendix B**.\n\n> Some experimental details are unclear. The paper should be carefully proofread.\n\n**A:** More details will be (have been) supplemented in the appendix file (such as detailed NLI scores calculation and complete hypotheses construction of relations) due to the **page limitation of a short paper**. We will definitely **release all related codes and data** to guarantee the reproducibility of our work. Please let us know if there still exists any uncleared details.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper proposed a novel relation triple generation method, which employs LLM to generate triples and uses NLI models to filter out generated triples.\n2. The experimental results show that the DocGNRE dataset has the ability to improve the recall of relation extraction, which is a critical evaluation for alleviating the false negative problem of the existing dataset.", "reasons_to_reject": "1. In Table 2, Precision and F1 should also be shown for Re and D - Re datasets. What is 'D - Re' and 'Re'?\n2. Some previous work also annotates the dataset. The evaluation of other annotated datasets should be conducted.\n3. The quality of the annotated datasets should be shown. Some human evaluations should also be conducted. Some data statistics should be shown.\n4. Some experimental details are unclear. The paper should be carefully proofread.", "questions_to_authors": "N/A", "paper_topic_and_main_contributions": "This paper aims to design an automated annotation method to alleviate the false negative problem of the document-level relation extraction (DocRED) dataset with minimum human effort. DocGNRE uses the large language model to annotate the relations among entities and employs NLI models to generate the entailment scores to estimate the quality of GPT-generated relation triples.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "line 282: improvedwith\nline 139: GPT-3.5(gpt-3.5-turbo)"}, "extra_text": {}}, {"review_id": "N7hddDRm5A", "reviewer": "Reviewer_y51e", "summary": "", "questions": "1 In line 060-062, what kind of datasets do you use to count the relation types? And \n2 It’s difficult for the reviewer to understand that vanilla in-context learning is infeasible due to relation types and LLMs. Why do relation types cause in-context learning problems, and what is the connection between LLM generation and in-context learning?\n3 In figure 1, in the P17, DocGNRE and Re-DocRED have similar numbers, but in P276, the two methods are very different. Does it mean that DocGNRE must have a wrong statistical distribution in P17 and P276? \n4 In section NLI Module as an Annotator, the author considers 192 possible hypotheses. But not all relationship types have an opposite relationship, more often there is NA. In addition, if the entity pair (A and B) has two relations (R1 and R2), the B and A may have the relation (R3). There is no opposite relations among R1, R2, and R3.\n5 Using NLI automatic generation, it is very likely that different entity pairs with the same relationship will generate exactly the same context. How to solve it?\n6 Why is the effectiveness of the method or dataset not verified experimentally using traditional methods? For example, ATLOP and KD-DocRE.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review and insightful comments. We provide detailed replies to your comments and hope we can resolve your major concerns.\n\n> In Lines 060-062, what kind of datasets do you use to count the relation types?\n\n**A:** We use the original document-level RE dataset DocRED to count the relation types. Note that both Re-DocRED and our DocGNRE are **re-labeling** DocRED, so **all the datasets have the same relation type set**.\n\n> It’s difficult for the reviewer to understand that vanilla in-context learning is infeasible due to relation types and LLMs. Why do relation types cause in-context learning problems, and what is the connection between LLM generation and in-context learning?\n\n**A:** Applying the currently popular in-context learning scheme in the Relation Extraction task to document-level Relation Extraction (docRE) faces two challenges: \n1. There are 96 different types of relations, and it's possible for multiple relations to exist between each pair of entities within the docRE task. So, both **designing precise instructions** to describe the task and **providing a comprehensive set of demonstrative examples** to effectively prompt and inspire the LLM to generate human-expected relations **are difficult**. \n2. The long context setting (document as input) of docRE, resulting in a **long prompt for LLM, hinders the LLM's ability to generate relations following human-given constraints** such as generating relations only from the given entity list and relation type set. What's more, there are generally **a large number of target relation triples of one document**. However, **the accuracy decreased as the generated content by LLMs became longer**. These phenomena may be caused by the weakness that LLMs still exist in understanding the long context. \n\nThe evidence is shown by Wadhwa, et al (2023) (<https://arxiv.org/pdf/2305.05003.pdf>) and our experiments. Wadhwa, et al (2023) have attempted experiments of in-context learning over DocRED and concluded \"`These results highlight a remaining limitation of in-context learning with large language models: for datasets with long texts or a large number of targets, it is not possible to fit detailed instructions in the prompt. In light of the issues we were unable to evaluate this approach on the DocRED dataset, which we leave for future work.`\" More details can be found in Section 3.3 of their paper. We conducted zero-shot docRE experiments and found pure GPT3.5 (gpt-3.5-turbo) only gets a 13.97 precision score on the full test set, which reflects that **lots of predictions of GPT3.5 are illegal given the pre-defined types** (maybe correct). More details can be found in Section 3.1 of our paper. \n\n> In figure 1, in the P17, DocGNRE and Re-DocRED have similar numbers, but in P276, the two methods are very different. Does it mean that DocGNRE must have a wrong statistical distribution in P17 and P276?\n\n**A:** Note that our dataset serves as an extension of Re-DocRED, which itself complements DocRED. So in Figure 1, DocGNRE contains Re-DocRED, while Re-DocRED contains DocRED. For P17, DocGNRE and Re-DocRED have similar numbers means that **the annotation of Re-DocRED in the P17 relation is sufficient, so our supplement is minor. For the P276 relation, we supplement a considerable number of relation triples**. \n\nAs demonstrated in Instruction of the paper, the reasons for the different numbers are that Re-DocRED is built by firstly pre-training a RE model with distantly supervised data from DocRED, then utilizing this model to predict triple candidates. So **the supplementary annotations in Re-DocRED exhibit a distribution similar to that of the DocRED test set** (DocRED contains a larger number of P17 relations than P276, such that Re-DocRED supplements more P17 relations than P276). \n\nHowever, we think such a process may **introduce model bias, potentially resulting in the exclusion of outlier relations that exist beyond the scope of the existing data distribution**. A similar observation also is illustrated by Huang et al. (2022) (<https://aclanthology.org/2022.acl-long.432.pdf>). That is why we re-labeling Re-DocRED and leveraging LLMs. **The large number of P276 relations we supplement proves that our method successfully achieves our goals and finds more 'out-of-distribution' correct triples**.\n\n> In the section NLI Module as an Annotator, the author considers 192 possible hypotheses. But not all relationship types have an opposite relationship, more often there is NA. In addition, if the entity pair (A and B) has two relations (R1 and R2), B and A may have the relation (R3). There is no opposite relations among R1, R2, and R3. \n\n**A:** We want to clarify the usage of 192 hypotheses. Recall that we take each generation triple $(e1, r_{gpt}, e2)$ from GPT as a premise, such as an example shown in Figure 2 of the paper: *<‘David Lean’, ‘worked for’, ‘London Films’>*. We want to find a predefined relation type from the relation type set (r1, r2, r3, ....) that corresponds to this triple. We construct 192 candidate hypotheses *[(e1, r1, e2), (e2, r1, e1), (e1, r2, e2), (e2, r2, e1), (e1, r3, e2), (e2, r3, e1), ...]*. Finally, through the NLI module, we only chose **ONE triple** with the **highest entailment score from the 192 candidate hypotheses** as our final prediction relation triple. \n\nThe reason why we consider 192 hypotheses rather than 96 is that **the relation is directional** and we want to match them more accurately. For example, if the predefined relation types contain \"employee\" rather than \"employer\", **<‘David Lean’, ‘worked for’, ‘London Films’>** will correspond to **<‘London Films’, 'employee', 'David Lean'>** rather than <‘David Lean’, ‘employee’, ‘London Films’>. So, **there are no problems when relations do not have opposite relations**.\n\n> Using NLI automatic generation, it is very likely that different entity pairs with the same relationship will generate exactly the same context. How to solve it?\n\n**A:** The NLI module takes the **GPT-generated triples concatenated with our designed hypotheses** as input and outputs the entailment scores. So, **there are no problems when different entity pairs with the same relationship**.\n\n> Why is the effectiveness of the method or dataset not verified experimentally using traditional methods? For example, ATLOP and KD-DocRE.\n\n**A:**  We conducted experiments using ATLOP and KD-DocRE methods by leveraging their officially provided code. We solely present the results of the DREEAM model because \n1. DREEAM is **the state-of-the-art model** and conducts **complete experiments** both on DocRED and Re-DocRED; \n2. Results of ATLOP and KD-DocRE show a **similar tendency to DREEAM**. \n\nThe results of ATLOP and KD-DocRE we conducted are shown in the following tables. \n\n**Results of ATLOP**\n|         |DocGNRE(D)|DocGNRE(D)|DocGNRE(D)| Re     | D - Re |\n| :-----: | :----:   | :----:   | :----:   | :----: | :----: |\n|Train set|     P    |     R    |    F1    |  R     |    R   |\n|DocRED $\\dagger$ |     87.9    |     28.6    |    43.2    |  31.9    |    0.9   |\n|DocRED $\\ddagger$ |     **89.8**    |     31.1    |    46.2    |  34.6     |    1.5   |\n|Doc + GPT $\\dagger$ |     82.9    |     29.3    |    43.2    |  31.2     |    13.0   |\n|Doc + GPT $\\ddagger$ |     84.4    |     32.2    |    46.6    |  34.4     |    13.3   |\n|Doc + mGPT $\\dagger$ |     79.6    |     30.2    |    43.8    |  31.5     |    19.54   |\n|Doc + mGPT $\\ddagger$ |     79.3    |     **34.2**    |    **47.7**    |  **35.7**   |    **21.1**   |\n|          |          |           |          |            |            |\n\n**Results of KD-DocRE**\n|         |DocGNRE(D)|DocGNRE(D)|DocGNRE(D)| Re     | D - Re |\n| :-----: | :----:   | :----:   | :----:   | :----: | :----: |\n|Train set|     P    |     R    |    F1    |  R     |    R   |\n|DocRED $\\dagger$ |     **82.4**    |     32.9    |    47.0    |  36.6     |    1.6   |\n|DocRED $\\ddagger$ |     76.9    |     37.8    |    50.3    |  41.9     |    2.7   |\n|Doc + GPT $\\dagger$ |     74.3   |     36.1    |    48.5    |  38.3     |    17.9   |\n|Doc + GPT $\\ddagger$ |     77.1   |     37.4    |    50.4    |  39.7     |    18.8   |\n|Doc + mGPT $\\dagger$ |     65.3    |     39.2    |    48.9    |  40.0     |    31.9   |\n|Doc + mGPT $\\ddagger$ |     68.8    |     **41.2**    |    **51.6**    |  **42.5**     |    **30.8**   |\n|          |          |           |          |            |            |\n\n$\\dagger$ represents BERT-base and $\\ddagger$ represents RoBERTa-large.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Well-written and well-motivated -Two kind of issues of document-level relation extraction should be important.", "reasons_to_reject": "It is not clear whether the approach could solve two issues. More experiments should be needed.", "questions_to_authors": "1 In line 060-062, what kind of datasets do you use to count the relation types? And \n2 It’s difficult for the reviewer to understand that vanilla in-context learning is infeasible due to relation types and LLMs. Why do relation types cause in-context learning problems, and what is the connection between LLM generation and in-context learning?\n3 In figure 1, in the P17, DocGNRE and Re-DocRED have similar numbers, but in P276, the two methods are very different. Does it mean that DocGNRE must have a wrong statistical distribution in P17 and P276? \n4 In section NLI Module as an Annotator, the author considers 192 possible hypotheses. But not all relationship types have an opposite relationship, more often there is NA. In addition, if the entity pair (A and B) has two relations (R1 and R2), the B and A may have the relation (R3). There is no opposite relations among R1, R2, and R3.\n5 Using NLI automatic generation, it is very likely that different entity pairs with the same relationship will generate exactly the same context. How to solve it?\n6 Why is the effectiveness of the method or dataset not verified experimentally using traditional methods? For example, ATLOP and KD-DocRE.", "paper_topic_and_main_contributions": "This paper tackles document-level relation extraction. Especially, the authors focus on designing an automated annotation method with minimum human effort. But there are some problems:  plenty of relation types and uncontrolled generations. To solve these problems, propose a method integrating a large language model ( LLM ) and a natural language inference (NLI) module to generate external relation triples to supplement document-level relation datasets. The experimental result shows that proposed method is effective.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "55AfnfGg8f", "reviewer": "Reviewer_rie9", "summary": "", "questions": "- On lines 148-151, it states, \"We notice that as the generated content by LLMs became longer, the accuracy decreased. To mitigate this, we set “at least 20 triples” in the initial prompt.\" I don't get this. If longer generations are less accurate, wouldn't the strategy be to ask the model to generate a _maximum_ of X triples?\n- On line 252, \"Pure GPT-3.5 only hits rare ground truth.\" What does this mean? Do you mean it rarely predicts ground-truth relations?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your constructive comments and acknowledgment of our novelty. We hereby provide a detailed response to address your concerns.\n\n> The paper makes some claims about its general applicability that I don't think are warranted. The method, as described, does not actually annotate more documents; it extends DocRED by identifying false-negative relations. It is therefore not likely to be useful for DocRE datasets that don't have the false negative problem that DocRED has. I think the authors would have needed to apply this to another DocRE dataset, or at least provide evidence that other DocRE datasets besides DocRED suffer similar false-negative problems, for me to buy the claim about this method's general applicability.\n\n**A:** First, the false negative problem naturally exists in many datasets, Han et al. (2023) (<https://arxiv.org/pdf/2305.14450.pdf>) evaluated ChatGPT on 14 sentence-level IE subtasks and indicated \"`unannotated spans is the most dominant error type. This raises concerns about the quality of annotated data, and indicates the possibility of annotating data with ChatGPT`\", which **reflects the false negative problems even in sentence-level tasks**. \n\nSecond, document-level relation extraction generally has a large number of entities and predefined relation types. As our demonstration in the Introduction, achieving complete manual annotation is challenging. Thus dataset construction generally starts with a recommend-revise scheme or a distant supervision. Previous works have shown that **both datasets with recommend-revise scheme** (Huang et al. (2022) <https://arxiv.org/pdf/2204.07980.pdf>) and **most distantly relation extraction datasets (containing sentence-level RE datasets)**, such as NYT and TACRED, **have false negative problems** (Hao et al.<https://arxiv.org/pdf/2109.02099.pdf>). \n\n> There are virtually no details given about the human evaluation (end of section 2.2). As I understand it, this is the final step before including a new triple into DocRED, so I would have expected much more details on the process, including some background on each annotator, whether they were involved in other parts of the study, any platforms used to perform the annotation, inter-annotator agreement, etc., etc.\n\n**A:** We use Mechanical Turk for human annotations. The annotators were not involved in other parts of the study. In order to ensure that annotators possessed a significant level of qualification, prospective annotators were required to meet the following criteria: \n- “HIT Approval Rate(%) for all Requesters' HITs” >95；\n- \"Number of HITs Approved\" > 1000；(the first two indicators are calculated by Mechanical Turk according to one's history performance)\n- \"Location\" is one of {United States, Canada, Great Britain, Australia, Singapore, Ireland, New Zealand} (to promise English proficiency).\n\nWe provided each NLI-annotated relation triple and its corresponding document context, alongside descriptive information regarding the relationship to annotators. Note that we proceeded with **two rounds of annotations**. In the first round, for each triple, two annotators were asked to determine its correctness based on the given context. In the second round, the third worker annotated the conflicting annotations of the first two workers. Due to the binary (yes/no) selection of the annotating, **the newly chosen triples for the test set must receive agreement from two annotators** who both deem them to be correct. **The acceptance rate of the final annotations is 71.3%**. \n\n\n> On lines 148-151, it states, \"We notice that as the generated content by LLMs became longer, the accuracy decreased. To mitigate this, we set “at least 20 triples” in the initial prompt.\" I don't get this. If longer generations are less accurate, wouldn't the strategy be to ask the model to generate a maximum of X triples?\n\n**A:** It is a trade-off between ensuring accuracy and the quantity of generated relations. We find that **the first 20 or so relations generated by GPT-3.5 (gpt-3.5-turbo) exhibit a relatively promising level of quality**. As for using \"at least\", we experimentally find that GPT-3.5 generates about 20 triples with a prompt containing \"at least\", whereas it generates only a small set of relations without it. For your suggestion, we made an effort to use the expression \"no more than\" and \"a maximum of\". We discovered that GPT-3.5 generates triples with comparable numbers and quality with all of these expressions.\n\n> On line 252, \"Pure GPT-3.5 only hits rare ground truth.\" What does this mean? Do you mean it rarely predicts ground-truth relations?\n\n**A:** Yes, it rarely predicts relations **exactly identical to** ground-truth relations. **Pure (without our NLI module)** GPT-3.5 (gpt-3.5-turbo) generates **most relations expressed by natural language**, which do not **exactly match** with the ground truth, even though some of these relations represent the same meaning as ground truth. So the exact-match F1 scores are unsatisfactory.\n\n> Suggestions\n\n**A:** Thanks for your thoughtful suggestions, we will polish our paper in the new revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Given DocRED's popularity, an \"enhanced\" or \"improved\" version could be a major contribution to the DocRE research community.\n- The proposed pipeline of LLM + NLI seems general enough that it could potentially be applied to other DocRE datasets, assuming they have similar issues of false negatives as DocRED.", "reasons_to_reject": "- The paper makes some claims about its general applicability that I don't think are warranted. The method, as described, does not actually annotate more documents; it extends DocRED by identifying false-negative relations. It is therefore not likely to be useful for DocRE datasets that don't have the false negative problem that DocRED has. I think the authors would have needed to apply this to another DocRE dataset, or at least provide evidence that other DocRE datasets besides DocRED suffer similar false-negative problems, for me to buy the claim about this method's general applicability. \n- There are virtually no details given about the human evaluation (end of section 2.2). As I understand it, this is the final step before including a new triple into DocRED, so I would have expected much more details on the process, including some background on each annotator, whether they were involved in other parts of the study, any platforms used to perform the annotation, inter-annotator agreement, etc. etc.", "questions_to_authors": "- On lines 148-151, it states, \"We notice that as the generated content by LLMs became longer, the accuracy decreased. To mitigate this, we set “at least 20 triples” in the initial prompt.\" I don't get this. If longer generations are less accurate, wouldn't the strategy be to ask the model to generate a _maximum_ of X triples?\n- On line 252, \"Pure GPT-3.5 only hits rare ground truth.\" What does this mean? Do you mean it rarely predicts ground-truth relations?", "paper_topic_and_main_contributions": "This paper proposes a method for augmenting existing document-level relation extraction (DocRE) datasets with additional annotations, using a combination of in-context learning (ICL) with large language models (LLMs) and a natural language inference (NLI) based filtering step. The process is used to create DocDNRE, a version of the popular DocRED corpus with an additional 2078 triples in the test set.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- The citation on line 028, after a general introduction of DocRE is a little strange, as it points to a specific methods paper in 2020 that didn't introduce the task. Instead of citing a _single_ paper here, I might cite a handful of popular DocRE papers. There are lots to choose from (see [here](https://www.semanticscholar.org/search?q=document%5C-level%20relation%20extraction&sort=total-citations))\n- Previous approaches using ICL + LLMs for DocRE should be cited, IMO. E.g. https://arxiv.org/abs/2212.05238\n- The overall approach of using NLI to filter the candidate relations feels reminiscent of similar pipelines in QA, e.g. https://arxiv.org/abs/2104.08731", "Typos_Grammar_Style_and_Presentation_Improvements": "__Typos, Grammar, Awkwardness in Writing__\n\nThere were quite a few typos and awkward spots in the writing, I would encourage the authors to take another pass. I provide some examples below:\n\n- On line 079 (and elsewhere) it should be \"in-context learning\" not \"in contextual learning\"\n- On lines 092-093 \"thus _enabling leveraging_ GPT to solve zero-shot document-level RE\", the word enabling should be dropped\n- The link to DocDNRE (in footnote 1) is broken (at least when I open the PDF). It redirects to https://anonymous/\n- On line 139 \"We have selected GPT-3.5(gpt-3.5-turbo)\" missing a space before the \"(\"\n- On line 282 \"improvedwith\" missing a space\n\n__Suggestions__\n\n- Why not use the human-readable names of relation types in Figure 1 (and elsewhere)? They could be shortened/abbreviated and may be displayed at a 45-degree angle if space is a concern\n- It is harder to understand the contributions of the paper than it should be. I would suggest a minor rewrite of the abstract, contributions section and conclusion. It would be helpful to clearly mention that you 1) introduce a method for re-labelling existing DocRE datasets, making it clear that it doesn't annotate new documents de-novo 2) mention the DocRED extension by name in the abstract, 3) Mention the number of new relations you added to the existing DocRED corpus, and list the relative increase\n- The final sentence in the abstract is a clear overclaim to me. I would re-work it.\n- I think the title kind of obscures the contributions of the paper a bit. The main problem involved is a semi-automatic way to recover missed relation annotations in existing DocRE datasets. The title doesn't capture this. In addition, I have never heard the phrase \"distant large language models\". Would recommend switching to \"distant supervision _with_ large language models\""}}]

Author rebuttal:

Meta-review: The objective of this paper is to devise an automated annotation approach aimed at ameliorating the false negative predicament in document-level relation extraction (DocRED) dataset with minimal human intervention. DocGNRE leverages a substantial language model for annotating relations between entities and incorporates Natural Language Inference (NLI) models to derive entailment scores, thus assessing the quality of the relation triples generated by GPT. Overall, this paper is robust. However, the authors are encouraged to enhance clarity in experimental specifications and certain notations in the revised version.
Decision: Accept-Main