DocKS-RAG: Optimizing Document-Level Relation Extraction through LLM-Enhanced Hybrid Prompt Tuning
Authors: Xiaolong Xu
Yibo Zhou
Haolong Xiang
Xiaoyong Li
Xuyun Zhang
Lianyong Qi
Wanchun Dou
Venue: ICML 2025 poster (2025)
Keywords: Document-level relation extraction
Structural knowledge integration
Semantic retrieval
Hybrid-prompt tuning
Primary area: deep_learning->large_language_models

Abstract: Document-level relation extraction (RE) aims to extract comprehensive correlations between entities and relations from documents. Most of existing works conduct transfer learning on pre-trained language models (PLMs), which allows for richer contextual representation to improve the performance. However, such PLMs-based methods suffer from incorporating structural knowledge, such as entity-entity interactions. Moreover, current works struggle to infer the implicit relations between entities across different sentences, which results in poor prediction. To deal with the above issues, we  propose a novel and effective framework, named DocKS-RAG, which introduces extra structural knowledge and semantic information to further enhance the performance of document-level RE. Specifically, we construct a Document-level Knowledge Graph from the observable documentation data to better capture the structural information between entities and relations. Then, a Sentence-level Semantic Retrieval-Augmented Generation mechanism is designed to consider the similarity in different sentences by retrieving the relevant contextual semantic information. Furthermore, we present a hybrid-prompt tuning method on large language models (LLMs) for specific document-level RE tasks. Finally, extensive experiments conducted on two benchmark datasets demonstrate that our proposed framework enhances all the metrics compared with state-of-the-art methods.

DocKS-RAG:
Optimizing Document-Level Relation Extraction through LLM-EEnanced Hybrid Prompt Tuning
Xiaolong Xu
Yibo Zhou'
Haolong Xiang'
Xiaoyong Li
Xuyun Zhang?
Lianyong Qi* Wanchun Dou S
Abstract
1. Introduction
Document-leve relation extraction (RE) aims to extract comprehensive correlations between en- tities and relations from documents. Most of existing works conduct transfer learning on pre- trained language models (PLMs), which allows for richer contextual representation to improve the performance. However, such PLMs-based meth- ods suffer from incorporating structural knowl- edge, such as entity-entity interactions. Moreover, current works struggle to infer the implicit rela- tions between entities across different sentences, which results in poor prediction. To deal with the above issues, we propose a novel and effec- tive framework, named DocKS- RAG, which in- troduces extra structural knowledge and semantic information to further enhance the performance of decument-level RE. Specifically, we construct Document-level Knowledge Graph from the ob- servable documentation data better capture the structural information between entities and rela- tions. Then, Sentence-level Semantic Retrieval- Augmented Generation mechanism is designed to consider the similarity in different sentences by retrieving the relevant contextual semantic infor- mation. Furthermore, we present a hybrid proompt tuning method on large language models (LLMs) for specific document-level RE tasks. Finally, ex- tensive experiments conducted on two benchmark datasets demonstrate that our proposed framework enhances all the metrics compared with otate-of- the-art methods.
Relation extraction (RE) a critical task in natural language processing (NLP) domain, with substantial applications in real-world scenarios, such as knowledge graph construction, information retrieval, and automated question answering (He et al., 2022; Xiong et al., 2017; Nayak & Ng, 2019). The goal of RE is to extract structured and semantic triplets in the form of subject - relation - object from the raw texts. With the rapid development of deep learning, sub- stantial efforts have been dedicated extracting the triplets within individual sentences, which is known sentence- level RE. Although researchers have achieved remarkable ad advancements in sentence-level RE (Zeng et al., 2021; Lee et al., 2020; Baldini Soares et al., 2019), conversely, the domain of document-level RE remains under-explored.
Document-level RE poses greater challenges than sentence- level RE counterpart, primarily due to the presence of nu- merous implicit relations that require relational reasoning for identification. Moreover, entities may be related across texts, paragraphs, or even chapters within a document, rather than relying solcly on extracting relations from isolated sentences, which requires comprehensive locument-level understanding and precise extraction. For example, as il- lustrated in Figure 1, the subjects "Wisconsin" and "the United States House of Representatives" along with the object "US", appear in disparate sentences, complicating the identification of relations between them.
Existing document-level RE methods mainly conduct transfer learning on pre-trained language models (PLMs) (Huguet Cabot & Navigli, 2021; Xu et al., 2021b; Jiang et al., 2022; Zhou al., 2020; Xu et al., 2025a), which focus the representation of entities to better capture the contextual information within the documents. However, such PLMs-based methods suffer from modeling the inter- actions between entities and relations, which are known as the structural knowledge. For example, from the semantic meaning of the relation "country"', we can infer that the entities within the extracted triplet related to locations. Ignoring such interactions hinders such PLMs-based meth- ods from capturing the inner correlations between entities and relations, and leads to poor performance of extraction, especially in the complex document-level RE tasks.
'School of software, Nanjing University of Information Science and Technology, China "College of Metcorology and Occanogra- phy, National University Defense Technology, China 'School of Computing, Macquarie University, Australia College of Com- puter Science and Technology, China University of Petroleum (East China), China SState Key Laboratory for Novel Software Technology, Nanjing University, China. Correspondence to: Hao- long Xiang <hixiang @ muist..du.cn.>.
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

DocKS-RAG: Optimizing Decument-Level Relation Extraction through LLM-EEhannee Hybrid Prompt Tuning
mantic Retrieval-Augmentedd Generation (SetRAG) mech- anism is designed consider the similarity of different sentences by retrieving the relevant contextual semantic in- formation. From the perspective of semantic aligmment, we transfer the structural knowledge from DocKG to informa- tive expressions, and generate the hybrid prompts with the retrieved semantic information from SetRAG. Furthermore, we present a hybriddpromph prommp tuning method, which conducts Chrameter-Efficiee Fine-Tuning (PEFT) (Hu et al., 2021) to enhance the adaptability and performance of LLMs on specific document-level RE tasks. Finally, extensive ex- periments are conducted on two open benchmark datasets, DocRED (Yao et al., 2019) and Re-DocRED (Tan et al., 2022a), to further evaluate the performance of our proposed framework.
James William Murphy (April 17, 1858-July I1, 1927) was a U.S. Representative. (...2 sentences ...) He was elected mayor Platteville for two-year term in 1904, and then elected the United States House of Representatives as a Democrat in 1906, defeating Joseph W.Baccock for the seat from Wisconsin's 's 3rd congressional district.
Subject: United States House of Representatives
Object: U.S.
Relation: country; applies to jurisdiction
Subject: Wisconsint
Object: U.S.
Relation: country; located in the administrative territorial entity
Figure 1. An example of cross-text relation extraction from raw documents.
The main contributions of this paper summarized as follows:
To incorporate the rich structural knowledge for better traction, several scholars consider constructing extra graphs to model the structural interactions within the documents, and then perform inference with the constructed graphs (Li et al., 2020; Nan et al., 2020; Qin et al., 2021; Zeng et al., 2020). Despite such graph-based methods pay more atten- tion the structural information, several limitations persist. Firstly, most of these methods lack of sufficient contextual information, which limits the semantic understanding of models. Moreover, most of such graph-based methods in- troduce extra noises and irrelevant connections, which will bring erroneous inferences and restrict the performance of extraction. Next, several works have been proposed to inte- grate pre-trained language models (PLMs) with additional knowledge graphs (KGs), aiming to enhance the models' global understanding across both structural and semantic spaces more effectively(Tang et al., 2020b; Fu et al., 2019; Sahu et al., 2019; Schlichtkrull et al., 2018; Xu et al., 2025b). However, PL are mostly trained on massive unstructured and contextual documentation data, while KGs are typically composed of structured and relational data. Such discrimi- nation leads the semantic misalignment between texts and graphs, which hinders PLMs from representing and utilizing structural knowledge within KGs, especially in complicated locument-level RE tasks.
We investigate the existing challenges in document- level RE task and propose novel framework DocKS- RAG, which improves the performance of LLMs in document-level RE by considering the extra structural and semantic information.
We design a sentence-level semantic Retrieval- Augmented Generation mechanism aimed at enhancing the semantic understanding of LLMs by retrieving the relevant sentences within the documents.
We design a hybrid prompts generation method to com- bine the relevant structural interactions with seman- tic information from the constructed document- knowledge graph and sentence-level knowledge base.
We conduct extensive experiments on two benchmark datasets. The experimental results show that our pro- posed framework consistently outperforms the compet- itive methods.
2. Related Work
2.1. PlMss-ased Methods for Document-Level Relation Extraction
Traditional pre-trained language models (PLMs). such as BERT (Devlin et al., 2019), have been widely applied in relation extraction (RE). REBEL (Huguet Cabot & Nav- igli, 2021) shows how RE can be simplified by expressing triplets as a sequence of text, and performs end-to-end RE for different relation types. Eider (Xie et al., 2022) transfers the attention from the entire documents to the subset of rele- vant sentences to improve both computational efficiency and extraction performance. RE-Flex (Goswami et al., 2020) tackles the challenge of unsupervised relation extraction by employing contextual matching. Later works achieve
To address the issues outlined above, we propose a novel and effective framework, named DocKS-RAG, which aims to improve large language models (LLMs), such as LLAMA3 and GPT-4 (Dubey et al., 2024; Achiam et al., 2023) to perform document-level relation extraction (RE) tasks by introducing additional structural knowledge and enriched semantic information. Specifically. to better capture the rich interactions between entities and relations, we construct a Document-level Knowledge Graph (DocKG) from the observable documentation data. Then, a Sentence-level Se-
2

---

DocKS-RAG: Optimizing Decument-Level Relation Extraction through LLM-Enhanced Hybrid Prompt Tuning
advancement by integrating the structural knowledge to im- prove the global understanding of such PLMs-based models. A-GCN (Qin et al., 2021) proposes attentive graph convo- lutional networks to improve the performance of RE by building the context graph. GAIN (Zeng et al., 2020) con- structs two extra graphs to capture the interactions among different mentions for accurate extraction. However, most of existing PLMs-based methods mainly focus on entity rep- resentations, while ignoring the rich interactions between entities and relations. Moreover, despite later works focus on introducing structural knowledge as priors, these meth- ods still suffer from the semantic misalignment between PLMs and graphs, which leads to poor performance on document-level RE tasks.
Formally, Given a document D comprising sequence of sentences S = {51, { .2,...., where s; represents the i-th sentence in the document, and the relation set R = fri,72.... P............................... ras is pre-defined. We aim to identify all the entities F = {ei, (2....,,k) and extract all the possi- ble triplets T from S. Each triplet 7 can be represented as (ea, ; es). where ea, 6t, E E are entities within the docu- ment, and I'm denotes the category of relation between these entities. The objective function can be expressed as:
M
L Σ
(abb log es/ ()) (1)
(E
where P represents the set of all unique pairs of entities (Pa, eb) identified from the document. Yabm is a binary indicator variable, which is set to 1 if the relation I'm exists between entities e, and eb, and 0 otherwise. P(ea, .. is the predicted probability that the relation I'm exists between the entities ea and cb. We aim to minimize the loss C between the predicted probabilities and the corresponding true labels to further identify relation among entity pairs.
2.2. Large Language Models for Document-Level Relation Extraction
With the rapid development of deep learning, large language models (LLMs) have revolutionized the natural language processing (NLP) domain (Chia et al., 2022; Soman al., 2023; Arsenyan et al., 2024). Recently, researchers have dedicated to utilizing LLMs for document-level RE tasks. AutoRE (Xue et al.. 2024) introduces a novel RE paradigm on LLMs, which eliminates the reliance on pre-defined re- lations and focuses on the triplet facts that are distributed across documents. GAP (Chen et al., 2024) proposes gen- erative context-aware prompt-tuning method to promote the adaptability of LLMs. GenRDK (Sun et al., 2024) generates extra auto-labeled data and employs denoising strategy for LLMs to enhance the extraction performance. KG-RAG (Soman et al., 2023) addresses the challenges of domain adaptation by optimizing the context extraction process, which allows LLMS to generate the accurate texts from the established knowledge. However, such newly proposed methods primarily leverage the powerful semantic under- standing capabilities of LLMs, while still largely ignoring the inherent heterogeneity between graph knowledge and the language models.
4. Methodology
4.1. Overview
In this section, we will introduce the details of our proposed DocKS-RAG framework, which is as shown in Figure 2. Firstly, we construct the Document-level Knowledge Graph (DocKG) with Graph Neural Networks (GNNs) to obtain both representations of entities and relations. Secondly, Sentence-level Semantic Retrieval-Augmented Generation (SetRAG) mechanism is designed to enrich the model's s contextual understanding gained from the relevant docu- ments. Finally, we present a hlyrid-prompp tuning method, which generates the combined prompts from DocKG and SetRAG, and integrates by conducting Frrameter-Elficienn Fine-Tuning (PEFT) to further improve the performance of LLMs on document-level RE tasks.
In general. previous PLM-based methods combat the ef- fective incorporation of interactions between entities and
4.2. Document-Leve Knowledge Graph Construction for Hybrid Prompts Generation
relations for better extraction. Currently, recent proposed LLM-based works still fail to alleviate the misalignment between structural knowledge and semantic information.
In this part, we construct document-level knowledge graph (DocKG) from the observable documentation data to obtain both entity and relation representations to further capture the topological information between them, which are presented in line 3 in Algorithm 1. Specifically, we employ GNNs, which are dominate approaches for constructing knowledge graphs (Li et al., 2020; Qin et al., 2021), to model the struc- tural interactions between relations set R and entities set E within the documents D).
3. Problem Definition
The primary objective for the general document-level RE task is to identify and classify the relations among entities mentioned throughout an entire document. Specifically, it aims to use annotated data from set of observable doc- uments Da, to train model C' that can accurately extract relations between entities in unobservable documents D.,
Entity and Relation Representations To overcome the heterogeneity with language model augmented entity repre- sentations, we focus on modeling the interactions between
where D, n D.- = O.
3

---

DocKS-RAG: Optimizing Decument-Level Relation Extraction through LLM-Ennhaced Hybrid Prompt Tuning
Stagel: Hy brid Prompts Generation from Docment-Level Knowledge Graph and Sentence-Level Semantic RAG
Document2
Pre-Defned Relations
head government
Document-Leve Knowledge Graph
Netrival
Decumear!
country citizenship place of hirth country
United State House of Represatatiess
Anno frilia deyphh
Mircsnsin
Reievant Salgraph
coumers
aoe aloodd sage
werk Saiiin
Jorhulctiin
diment's Lobtut fonen of Uhis Actirtog Aveph IP, Dehoott resprnsienal Alokity-))
Triplets
Mlace
boraue
Hybrid Prompts
Hiprenen du suhen the Cnntud Sewee
...
anslics jutisdation,
Juses William Marpat
graduate
the University of
States Eud)
Observable Document Sets
Mickigos Law Nehioo
Erptod rechepuorty crtor'
Sepuenntation
Segmented Sentence Sets
Sentence-Lond Llvel Knowlelge Base
[1] James Murply (April
U.S. Representative
Stored
Informative Generation
[2] Marphy geadunted from the State Normal School 1873 University Michigan School 1880.. [31 He practiced ed Platteville, and verved av district atorney Grant Countr, If Riccossi from 1887 1891...
Seatcnes
iPLM Embesking
Senteacs
Frompts Extemion
Stage2: Hybrid- Prompt Tuning on Large Language Models for Document-Love Relation Extraction
Parameter-Efficceu Taning on Large Language Model
Avnry Hillin Maplly (Apil 1828-Jely Ut, 1927) 6.S Rigrosniatree (.)
L
Attention
from Miccomin of United Stating The fall assu of U.S United Stasing (-)
Sensaric promens from Selt.0 sabeepuentl
14-
Inpat
Fine- Tuned LLM
expert autuntion aared
Jovrn
Wo
Reyosus:
ellicsnnns, country. Regrosenttions
Hidden States
(..
Figure Overview of our proposed DocKG-RAG framework. contains two key stages: (1) Hybrid prompts generation from both document-level knowledge graph (DocKG) and sentence-level semantic RAG (SetRAG); (2) Hybrid-promp tuning based on large language models (L.LMs) with LORA.
entities and relations to better capture the contextual infor- mation within the documents. Formally, in a general graph G(E,, R). each entity e E is represented as an embedding vector h E Rd, where d is the dimension of the embed- ding size. Similarly, relations are represented by vectors that are embedded as h Idd. Then, we employ GNNs for document-level graph modeling and representation learning- In the training stage, entity and relation representations are iteratively updated by aggregating the neighbours' informa- tion. The update rule at layer is represented as:
where (ee) represents the set of neighboring entities of W. is a learnable weight matrix of the entity embedding, b denotes a learnable bias vector, and o referred to a non-linear activation function. W, is the learnable weight matrix for the relationship embedding, and the summation is over all pairs of entities and with relationship in the set E. The agg function aggregates the embeddings h, and h, of the entities at layer 1 1
Document- Knowledge Graph Construction and Relation Prediction. After L rounds training with GNNs, the representations are utilized for relation prediction be- tween the entities. For a pair of entities (ea, eb), we compute their interaction scores follows:
h
Σ c'EN(e)
Wohe- (1-1)
+b).
I(ea, eb) h (L)
he. h
(L)
h(()
(4)
(1)
W, + Σ
agg(h (-11) h bQ-1))
where dot-product is conducted between entity representa- tion (L.)) and (L) and (L is the representation for the
(uN)EE

---

DocKS-RAG:
Optimizing Decument-Level Relation Extraction through LLM-Enhanced Hybrid Prompt Tuning
Algorithm 1 Hybrid-Promps Tuning on Large Language
spaces between PLM and DocKG, in our proposed method, integrate PL.M embeddings as initial inputs for entities and relations, allowing DocKG to learn information from the PLM through the process of updating and improve the reliability of similarity calculations. The semantic transfor- mation is represented as:
Models
1:
Input: Observable documents Du. Given triplets Tu. Graph model Gm Embedding model Em Informative generation function Gen, Recycling times T, Parame- ters of adapter A, User query Q.
2: Output: Predicted triplets Tp-
hg = (q)
3: KG + Train (Gm,Tiy Tu)
4: Segment D, into individual sentences and obtain the
Then, for each entity e E E, we further compute the similar- ity between h. and h with a similarity function sim(:.)), which calculated by cosine similarity as follows:
sentence sets S.-
5: KB t Em (S.)
6: Retrieve the relevant subgraph G2 and Topk sentences
sim(g, e) = hg he
Ss from KG and K B according Q.
(8)
7: Pg < Gen (G,), P. Gen (Sx)
ba hell
8: Ph t- Concat (g.
Based on the computed similarities, we select the most relevant entities from DocKG and construct the candidate subgraph Gg as:
9: for f 1 to T do
10: LLM t- Train (LLMata, Pr, A)
11: Update parameters of adapter A
G, - sim Teg
12: end for
T' < LLM (Ph)
where Teg is pre-defined threshold that determines the retained sets of entity and relation interactions that most relevant with the query q. Then, we create the informative prompts using the structural interactions within the retrieved subgraphs, along with generation function JDecKG, and the process of the informative prompts P PDecKG generation is represented as:
14: Return: Т
relation between ea and eb. Then, we employ a softmax function to calculate the predicted probability of the relation type ry between ea and eb. the process of calculation is formulated as:
DocKG = DDocko relt)rrGG))).......)).)))
(10)
exp(I(en Eb, r,,r)))
P(rylea,es) eb) EnER exp(eexp(Tr exp(I(eax Cb. rk))
(5)
By adopting the proposed DocKG retrieval method, we ef- fectively exploit the structural interactions between different entities and relations. Moreover, the relevant subgraphs as- sociated with the user query are efficiently extracted and transferred into informative prompts, which further miti- gates the semantic misalignment between graphs and texts.
Finally, we construct the document-level knowledge graph, the construction of DocKG processed as:
DocKG = {Ce, r, ex) P(rylec et) > Ter),
(6)
where Ter is pre-defined threshold to determine the mini- mal probability of relation between entities ea and es-
4.3. Sentence-Level Semantic RAG Enhancement
In this part, to further enhance the semantic understanding of LLMs for document-level RE, we design a sentence-level semantic Retrieval-Augmeente Generation (SetRAG) mech- anism, which are presented in lines 4 to 5 in Algorithm 1. It mainly involves three processes for documents: segmen- tation, embedding and matching, Specifically, individual sentences are firstly segmented from the raw texts. Then, we construct the sentence-level knowledge base (SetKB) by embedding such sentences to the vector representations. Finally, relevant contextual sentences in the given document D will be retrieved by similarity measures and incorporated into the hybrid prompts.
In this way, we effectively leverage GNNs to capture the rich interactions between entities and relations within different documents. Meanwhile, the constructed DocKG consists of comprehensive representations in the semantic space, which enables more accurate extraction for document-level RE.
Graph Retrieval for Hybrid Prompts Generation. To efficiently extract structural knowledge from DocKG, we employ graph retrieval method for further hybrid prompts generation. The objective of graph retrieval identify relevant subgraph G3 C DocK G' associated with the given query q, and then obtain the informative prompts PDocKG based on the retrieved subgraph.
Sentence Segmentation. For a given document D, we aim to segment it into individual sentences. The process of segmentation is formulated as:
Specifically, we firstly transform the query into the se- mantic vector representation hg by pre-trained language model (PLM). Due to the differences in the embedding
Segment( D) = {81, 82, 82,...,
(11)
5

---

DocKS-RAG: Optimizing Decument-Level Relation Extraction through LLM-Enhanced Hybrid Prompt Tuning
where S' is the set of segmented sentences, and is the total number of extracted sentences. The segmentation function Segment utilizes punctuation (e.g., periods, exclamation marks) and linguistic rules to identify sentence boundaries.
Hybrid Prompts Generation. For a query 4, we generate the hybrid prompts P incorporating both relevant structural and semantic information from DocKG and Se- tRAG. The construction of hybrid prompts is expressed as:
Sentence Embeddings and Knowledge Base Construc- tion. After we obtain the segmented sentences set, we further propose a sentence- knowledge base (SetKB), which is constructed resent the semantic meaning of sentences. Specifically, each sentence 8; E S is embedded into a vector representation by a pre-trained language model (PLM), the construction of S is processed as:
PHybrid = Concat (PDocko(q), Pssanac(a)). (16)
collection of embeddings that rep-
where PDocka(g) represents the relevant structural knowl- edge retrieved from DocKG, and PSseRAG(()) represents the Topk relevant semantic information retrieved from Se- ARAG. The specific calculations are referred to Equs.(7-10) and Equs. (13-15).
= PLMM(s (s:), SetKB ((5, ti) i n), (12)
Parameter-Efficee Fine-Tuning LLMs. As konw, LLM is initially fine-tuned on large, annotated dataset and consists of massive generalized knowledge (Goswami et al., 2020). Subsequently, In our proposed DocKS-RAG framework, we adopt LORA, a general adapter training method (Hu et al., 2021), to fine tune LLMs. Specifically, by integrating adapter modules, we are supposed to achieve fine-tuning without modifying the entire parameter set of the pre-trained LLM. Moreover, the adapter modules A are trained on smaller, task-specific datasets on document-level RE, which allows for rapid adaptation and minimizes the computational complexity of our proposed framework. The output with PEFT is calculated as:
where each sentence 9; is converted into its semantic em- bedding vector ti and stored in the constructed S.
Knowledge Base Retrieval Process. The constructed SetKB enables semantic retrieval of relevant sentences as- sociated with the user queries. Specifically, given a query q. the most relevant sentences are identified and retrieved through similarity measuring. The calculation of measuring is formulated as:
((g, t;) = sim (LL(()). ta)). SetKB, (13)
where -)m is similarity function. The topk relevant sentences from the SetKB are then retrieved. The process of the retrieval is expressed as:
OPEFT = LLM (Plybridd 0 A))
(17)
S Topk (Mgp(MMg,.ea (M k.)
(14)
where @ are the original parameters of the pre-trained LLM, and A is referred to the learnable parameters associated with the adapter module that will be fine-tuned.
Finally, we further generate another part of hybrid prompts PSEIRAG using the retrieved sentences from SetKB, Sg. and theirs relevant triplets Te. The generation of PSetRAG is represented as:
In summary, we propose novel and effective framework, named RocKS-RAG, which mainly consists two stages.
In stage 1, we aim to generate the informative hybrid prompts by combining the relevant retrieved document- level structural knowledge and rich semantic information, which are based on the constructed document-level knowl- edge graph and sentence-level knowledge base. In stage 2, we consider conducting
PSERAG = ySaRAG ( (T.T. E T&)) + Se-
(15)
The integration of sentence-level semantic RAG enhances the LLM's ability to accurately extract relations by pro- viding rich and relevant semantic information, which em- powers the semantic understanding of LLMs and ultimately improves the performance of document-level RE
efficient fine-tuning on LLMs with the extended hybrid prompts to further enhance the adaptability and performance of LLMs on the complex document-level RE tasks.
4.4. LLM--nhhaned Document-Leve Relation Extraction through Hybrid Prompt Tuning
5. Experiments
In this part, we will give the detailed description of our designed hybrid-prompt tuning framework that leverages both document-level Knowledge Graph and sentence-level Generation (RAG) and con- semantic Efffciieer- Fine-Tuning (PEFT) to enhance ducts Parameter- the performance of LLMs on document-level RE tasks, which are presented in lines 6 to 12 in Algorithm
5.1. Datasets.
We evaluate the performance of our proposed DocKS-RAG on two widely used benchmark datasets: DocRED (Yao et al., 2019) and Re-DocRED (Tan et al., 2022a). We refer the reader to Appendix A.1 1 for more detailed description of both the datasets.
6

---

DocKS-RAG: Optimizing Decument-Level Relation Extraction through LLM-Enhance Hybrid Prompt Tuning
Table Comparison of performance DocRED and Re-DocRED datasets, showing both development (Dev) and test (Test) results with genFFl and Fl scores. The best results are in bold and the second best is underlined . Our proposed DocKS-RAG outperforms other methods on both datasets.
DockED
Re-DocRED
Model
Dev
Test
Dev
Test
Ign-F1
F1
Ign-F1
F1
Ign-F1
F1
Ign-F1
F1
GEDA (2020)
54.52
56.16
53.71
55.74
LSR (2020)
52.43
59.00
56,97
59.05
GAIN (2020)
59.14
61.22
59.00
61.24
71.99
73.49
71.88
73.44
HIN (2020)
54,29
56,31
53,70
56.60
HeterGSAN (2021)
58.13
60.18
57.12
59.45
ATLOP-BERY (2021)
59.22
61.09
59.31
61.30
73.35
74.22
73.23
74.02
SSAN (2021)
56.68
58,95
56.06
58,41
SIRE (2021)
59.82
61.60
60.18
62.05
DocuNet (2021)
59,86
61,83
59.93
61.86
73.60
74.62
73.53
74.48
KMGRE (2022)
73.33
74.24
73.39
74.46
KD-DocRE (2022)
60.08
62.03
60.04
62.08
73.68
74.66
73.64
74.55
SRLR (2023)
60.02
62.14
59.74
61.88
AutoRE (2024)
54.29
53.84
LMRC (2024)
59.22
61.08
58.62
60.69
72.35
72.93
DocKS-RAG(Ours)
62.76
63.87
61.83
63,59
74.64
75.31
74.32
75.49
5.2. Baselines.
5.5. Results Analysis
We evaluate our proposed framework by comparing with recently presented PLMs-based, graph-based and LLMs- based methods, including HIN (Tang et al., 2020a), A BERT (Zhou al., 2020), SRLR (Huang et al., 2023), GLRE (Wang et al., 2020), LSR (Nan et al., 2020), Het- erGSAN (Xu al., 2020), SIRE (Zeng al., 2021), SSAN (Xu et al., 2021a), DocuNet (Zhang et al., 2021), KD- DocRE (Tan et al., 2022b), KMGRE (Jiang et al., 2022), AutoRE (Xue et al., 2024), and LMRC (Li et al., 2024). We refer the reader to Appendix A.2 for more details of the baselines.
Document-Level Relation Extraction. Table summa- rizes the performance of DocKS-RAG comparing with var- ious methods on DocRED and Re-DocRED datasets, and the advanced conclusions are summarized as follows:
Firstly, comparing with previous PLMs-based methods, conduct graph retrieval extra constructed document- level Knowledge Graph (DocKG) to better capture the struc- tural knowledge within the documents. We later generate the informative prompts from the retrieved subgraphs, which mitigates the heterogeneity between structural graph knowl- edge and the language models. Secondly, comparing with recent proposed graph-based methods, we consider the simi- larity between sentences, and further design sentence-level semantic Retrieval-
5.3. Implementation Details.
The technical modules of DocKS-RAG primarily consist of document-Lewe knowledge graph construction, sentence- level knowledge base construction, and hybrid-prompt tun- ing. The parameters for the comparative experiments are as reported in their paper. We refer the reader to Appendix A.3 for more details of the experimental parameter settings.
Generation (SetRAG) mech- anism to enrich the contextual understanding of models. Furthermore, we present a hybrid-prompt tuning method, which generates the hybrid informative prompts by retriev- ing the relevant information both from DocKG and SetRAG, integrating with an adapter training method further enhances the adaptability of LLMs for better extraction. Finally, the experimental results demonstrate that proposed DocKS- RAG outperforms competitive methods in both Ign-F1 and Fl score metrics on all datasets, which indicates the effec- tiveness of our framework on document-leved RE tasks.
5.4. Metrics.
In general document-leved RE tasks, both the Fl and Ign-Fi Fl scores play crucial roles in evaluating document-level RE methods. The F1 score allows for a balanced assessment, where both false positives and false negatives are taken into account. On the other hand, the Ign-Fl score (Ignored score) provides a clearer picture of a model's 's effectiveness in extracting the implicit triplets from the documents. In this paper, employ both F1 and Ign-F1 scores our evaluation metrics, where higher values indicate better per- formance in document-level RE.
Document-Level Knowledge Graph Retrieval. To fur- ther investigate how the document-level knowledge graph (DocKG) retrieval effects the performance of LLMs on document-level RE tasks, we conduct extra experiments on the development set of RE-DocRED with different Teg E [0, 1]. which is a pre-defined threshold that for rel- evant subgraphs retrieval. As illustrated in Table 3. Teg

---

DocKS-RAG: Optimizing Decument-Level Relation Extraction through LLM-Enhanced Hybrid Prompt Tuning
Table Ablation study results both development (Dev) and test (Test) results with Ign-Fi and F1 scores. The performance declines the removal of different components.
DocRED
Re-DocRED
Model
Dev
Test
Dev
Test
Ign-F1
F1
Ign-F1
F1
Ign-F1
F1
Ign-F1
F1
DocKS-RAG
62.76
63.87
61.83
63.59
74.64
75.31
74.32
75.49
w/o DocKG
58.21
58.89
57.26
58.08
68.17
68.59
68,81
69.03
w/o SetRAG
55.09
53,51
54.32
54.96
61.64
62.37
62.29
62.63
w/o Elybrid-Prompp Tuning
20.46
21.07
20.53
20.94
26.15
26.87
25.89
26.32
to the decline of overall performance and model reliability.
Table 3. Performance comparison with different threshold Tag in DocKG development (Dev) of Re-DocRED. Experimental
results achieve peak performance at Teg = 0.7.
5.6. Ablation Study
Ours-DocKG
RE-DocRED (Dev)
To assess the effectiveness of each component in our DocKS- RAG framework, we further perform an ablation study by removing different components step by step. As illustrated in Table 2, DocKS-RAG with all components, achieves the highest performance, showcasing the positive contribu- tion of cach component in our proposed framework. As the document-level knowledge graph (DocKG) component is removed, the model's 's performance declines, which suggests that the integration of structural knowledge is crucial for LLMs to capture the structural information between enti- ties and relations. Then, the removal of the sentence-level semantic Retrieval-Augmented Generation (SetRAG) com- ponent results in further performance degradation, which demonstrates that our designed SetRAG mechanism has sufficient influences in enhancing the semantic understand- ing of LLMs for better extraction. Moreover, such results demonstrate the necessity of our proposed hybrid prompts generation method, which effectively alleviates the misalign- ment between structural knowledge and semantic informa- tion. Furthermore, the removal of Hybrid-Prompt Tuning leads to drastic reduction in performance than others. In our framework, such case equals to Few-shot learning on LLMs, which means the model only relies on transferred knowledge from few instances and leverages such experi- ence for new tasks. The results shows that few-shot learning method is not suitable for complex document-level RE tasks in our framework, and underscores the necessity of our proposed hybrid-prod prompt tuning in optimizing the model's s expressiveness and accuracy.
Teq
0.2
0.5
0.7
0.9
F1 36.42
53,26
67.45
75.31
73.89
68.59
Ign-FI
32.06
49.51
63.37
74.64
73.16
68.17
Table 4. Performance comparison with different threshold & in SetRAG on development (Dev) set Re-DoCRED Experimental results achieve peak performance at k =
Ours-SetRAG
RE-DocRED (Dev)
0
10
F1
67.78
70.15
74.26
75.31
71.23
68.26
Ign-F1
65.32
68.54
72.75
74.64
71.52
68.45
has critical influence on the performance of DocKG-RAG. Specifically,, at Tey = 0, DocKS-RAG exhibits the worst performance, indicating that all entities in DocKG are con- sidered relevant to those within the user queries. Such case equals to randomly selecting an entity from DocKG and constructing its corresponding subgraph, which will bring massive noise and lead to poor prediction. As Tey increases, both F1 and Ign-Fi scores show substantial improvement and achieve peak performance Tag = 0.7, reflecting the effectiveness of graph retrieval by capturing the rich inter- actions between entities and relations. However, the per- formance declines when the threshold raises more, which demonstrates that only considering the most relevant entitiy in DocKG limits the global understanding of model.
Sentence-Level Knowledge Base Retrieval. We further in- vestigate how Set mechanism effects the performance of LLMs on document-level RE tasks. Extra experiments are performed on the development set of RE- DE-DDoRRED with different & E N. which is pre-defined threshold that means the Topk relevant sentences will be retrieved. As demon- strated in Table 4, the parameter has sufficient influence in the performance of DocKS-R. -OAGS As the threshold increases, both Fl and Ign-Fl scores exhibit significant improvements, reaching peak performance at A = 5, which shows that retrieving relevant contextual sentences is helpful to im- prove the relational and semantic understanding of LLMs for document-level RE tasks. However, as the threshold in- creases more, lowly relevant or even irrelevant sentences are probably retrieved, which will bring extra noises and lead
6. Conclusion and Future Work
In this paper, we propose a novel and effective framework, named DocKS-RAG, which enhances the adaptability and performance of LLMs for complex document-level RE tasks. Extensive experiments on two benchmark datasets demon- strate that our proposed DocKS-RA significantly outper- forms state-of-the-art methods. In the future, we will ex- plore the following problems on how to apply DocKS-RAG to different domains, and how to enhance the scalability of DocKS- -AG with different scales of LLMs.
8

---

DocKS-RAG: Optimizing Decument-Level Relation Extraction through JLM-Enhanced Hybrid Prompt Tuning
7. Response to Reviewers
7.3. Response to reviewer 3:
7.1. Response to reviewer 1:
Q1: The authors need to explain why they believe graph- based methods lack contextual information and what spe- cific types of context missing.
QI: Could the authors elaborate on the trade-offs between complexity and efficiency in practical deployments?
Response to QI: Graph-based methods primarily focus on the structural relationships between entities, which can obscure essential semantic nuances due to the static nature of graph representations. In complex document-leve relation extraction tasks, entities may relate across texts, paragraphs, or even chapters within document. This limitation in the graph- based approach restricts our understanding of how entities interact in different contexts, ultimately hindering the effectiveness of extraction.
Response Q1: In practical applications, we take full advantage of the parallel processing capabilities provided by both DocKG and SetRAG, enabling us to handle tasks efficiently while maintaining high performance of relational extraction. Additionally, by applying threshold retrieval strategies, we could further simplify the complexity of our operations.
Q2: How DocKS-RAG specifically mitigates the noise dur- ing knowledge graph construction and retrieval processes?
Q2: In (10), the authors introduce a generation function G.oeKG How do they implement this function?
Response to Q2: As presented in Section 4, firstly, DocKS- RAG applies a threshold parameter Ter during graph re- trieval to filter out extraneous entities. Secondly, another threshold parameter Teg is used to retrieve the relevant con- textual sentences. Finally, DocKS-RAG generates hybrid prompts that combine the information from both the DocKG and SetRAG, which integrates relevant contextual informa- tion to enhance semantic alignment, improving the overall extraction performance.
@DocKG is implemented as function for graph knowledge transformation. We pre-defined various entity-relation mapping types. For example, if a subgraph has "Wisconsin" and "U.S." connected by the edge "state", it would be transformed into the statement "Wisconsin is state of "U.S." based on our mappings. Ultimately, we concatenate such statements with the relevant semantic in- formation retrieved from SetRAG, and obtain the hybrid prompts.
Response Q2:
7.2. Response to reviewer 2:
CI: It might be useful to briefly explain how SetRAG differs from typical retrieval mechanisms.
7.4. Response to reviewer 4:
Q1: Could you provide detailed explanation of the opera- tions of Formula 10 and Formula 15?
Response to Cl: Our proposed SetRAG module constructs a knowledge base from segmented sentences and retrieves contextually relevant information based on embeddings, al- lowing for improved understanding of the document's ss se- mantics. Such semantic information will be integrated with structural knowledge from DocKG, which further enhances the performance of LLMs in relation extraction tasks.
Response to Q1: As illustrated in Section 4. the purpose of Formula 10 is to generate informative prompts based on the retrieved entities and relations from DocKG. It defines the prompt generation function gDocKG which takes a set of relevant entities and relations input and outputs structured prompts PDcKKG. Formula 15 constructs hybrid prompts using the retrieved sentences from SetKB. It employs the generation function ISe to create prompts from the relevant triplets T' while appending the retrieved sentences Sy. This integration enriches the prompts by combining contextual insights with structured knowledge, enhancing the model's 's ability to perform document-level relation ex- traction accurately.
Q1: How do the authors explain that simpler configurations that do not utilize DocKG or SetRAG yield reasonably good scores?
Response to QI: As shown in Table 4, the experimental results indicate that the integration of the DocKG and the SetRAG mechanisms significantly enhances model perfor- mance, demonstrating the effectiveness of our proposed DocKS-RAG module. practical deployment scenarios, we capitalize on the parallelization capabilitiess offered by both DocKG and SetRAG, which allows for efficient pro- cessing without compromising the quality of relational ex- traction. Furthermore, by implementing strategies such as threshold retrieval, we manage to streamline the complexity of the operations involved. Integrating both modules en- hances the model's adaptability to various document types and ensures robust performance in real-worh applications.
Q2: Can you claborate on how your framework specifi- cally address the semantic misalignment between PLMs and knowledge graphs?
Response to Q2: As mentioned in Section 4.2, the embed- ding spaces of PLM and DocKG differ significantly. To address this issue, we integrate PLM embeddings as the initial inputs for entities and relations, which allows DocKG to learn from the PLM during the updating process and improves the reliability of similarity calculations.

---

DocKS-RAG: Optimizing Decument-Level Relation Extraction through LLM-Enhaneed Hybrid Prompt Tuning
Acknowledgments
Fu, T.-J., Li, P.-.., and Ma, W.-Y. GraphRel: Modeling text relational graphs for joint entity and relation extraction. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pp. 1409-1418 1418, 2019.
This work was supported in part by the Jiangsu Provin- cial Major Project on Basic Research of Cutting-edge and Leading Technologies, under grant no. BK20232032 and National Natural Science Foundation of China under grant no. 92267104.
Goswami, A., Bhat, A., Ohana, H., and Rekatsinas, T. Unsu- pervised relation extraction from language models using constrained cloze completion. In Findings of the Associ- ation for Computational Linguisties: EMNLP 2020, pp. 1263-1276, 2020.
Impact Statement
This paper presents work whose goal to advance the field of Deep Learning. There are many potential societal conse- quences of our work, none which we feel must be specifi- cally highlighted here.
He, J., Zhou, C.,, Ma, X., Berg-Kirkpatrickk T., and Neubig, G. Towards unified view of parameter-efficien trans- fer learning. In International Conference on Learning Representations, 2022.
References
Achiam, O. J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I,, Aleman, F. L.., and Diogo Almeida, e. a. Gpt-4 techni- cal report. 2023.
Hu, J. E., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., and Chen, W. Lora: Low-rank adaptation of large language models. ArXiv, abs/2106.09685, 2021.
Huang. H., Yuan, C., Liu, Q.. and Cao, Y. Document-level relation extraction via separate relation representation and logical reasoning. ACM Transactions on Information Systems, 42:1 224, 2023.
Arsenyan, V., Bughdaryan, S., Shaya, F., Small, K. W., and Shahnazaryan, D. Large language models for biomedical knowledge graph construction: Information extraction from EMR notes, In Proceedings of the 23rd Workshop Biomedical Natural Language Processing, pp. 295-
Huguet Cabot, P.-L. and Navigli, R. REBEL: Relation extraction by end-to-end language generation. Find- ings of the Association for Computational Linguisties: EMNLP 2021, pp. 2370-2381, 2021.
317, 2024.
Baldini Soares, L., FitzGerald, N., Ling, J., and Kwiatkowski, T. Matching the blanks: Distributional similarity for relation learning. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguisties, pp. 2895-2905, 2019.
Jiang, F., Niu, J., Mo, S., and Fan, S. Key mention pairs guided document-leved relation extraction. In Proceed- ings of the 29th International Conference on Computa- tional Linguistics, pp. 1904-1914, 2022.
Chen, Z., Li, Z.,, Zeng, Y., Zhang, C., and Ma, Gap: novel generative context-awand prompt-tuning method for relation extraction. Expert Syst. Appl., pp. 123478, 2024.
Lee, S., Seo, S., Oh, B., Lee, K.-H., Shin, D., and Lee, Y. Cross-sentence n-ary relation extraction using entity link and discourse relation. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, CIKM '20, pp. 705-714, 2020.
Chia, Y. K., Bing, L., Poria, S., and Si, L. RelationPrompt: Leveraging prompts to generate synthetic data for zero- shot relation triplet extraction. In Findings of the Asso- ciation for Computational Linguistics: ACI. 2022, pp- 45-57, 2022.
Li, B., Ye, W., Sheng, Z., Xie, R., Xi, X., and Zhang, S. Graph enhanced dual attention network for document- level relation extraction. In Proceedings of the 28th In- ternational Conference on Computational Linguistics, pp. 1551-1560, 2020.
Devlin, J., Chang, M.-.., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for lan- guage understanding. In Proceedings of the 2019 Confer- ence the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, Volume (Long and Short Papers), pp. 4171-4186, 2019.
Li, X., Chen, K., Long, Y., and Zhang, M. Llm with relation classifier for document- relation extraction. ArXiv, 2024.
Nan, G., Guo, Z., Sekulic, I., and Lu, W. Reasoning with latent structure refinement for document-level relation extraction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1546 1557, 2020.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., and Akhil Mathur, e. a. The llama 3 herd of models. ArXiv, abs/2407.21783. 2024.
10

---

DocKS-RAG: Optimizing Decument-Leve Relation Extraction through LLM AJL--nnheced Hybrid Prompt Tuning
Nayak, T. and Ng. H. T. Effective modeling of encoder- decoder architecture for joint entity and relation extrac- tion. In AAAI Conference Artificial Intelligence, 2019.
Wang, D., Hu, W. Cao, E., and Sun, W. Global-to-local neural networks for document-level relation extraction. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), PP.
Qin, H., Tian, Y., and Song, Y. Relation extraction with word graphs from n n-grams, In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 2860-88628820 2868, 2021.
3711-3721, 2020.
Xie, Y., Shen, J., Li, S., Mao, Y., and Han, J. Eider: Empow- cring document-level relation extraction with efficient evidence extraction and inference-stage fusion. In Find- ings of the Association for Computational Linguisties: ACL 2022. pp. 257-268, 2022.
Sahu, S. K., Christopoulou, F., Miwa, M., and Ananiadou, S. Inter-sentence relation extraction with document-level graph convolutional neural network. In Proceedings of the 57th Annual Meeting of the Association for Compula- tional Linguistics, pp. 4309 40109431662220 2019.
Xiong, C., Power, R., and Callan, J. Explicit semantic rank- ing for academic search via knowledge graph embedding. In Proceedings of the 26th International Conference on World Wide Web, WWW '17, pp. 1271-1279, 2017.
Schlichtkrull, M., Kipf, T. N., Bloem, P., van;den Berg. R., Titov, I., and Welling, M. Modeling relational data with graph convolutional networks. In The Semantic Web: 15th International Conference, ESWC 2018, Heraklion, Crete, Greece, June 3-7, 2018, Proceedings, pp. 593-607, 2018.
Xu, B., Wang, Q., Lyu, Y., Zhu, Y., and Mao, Z. Entity structure within and throughout: Modeling mention de- pendencies for document-level relation extraction. In AAAI Conference on Artificial Intelligence, 202
Xu, W., Chen, K., and Zhao, T. Document-level relation extraction with reconstruction. In AAAI Conference on Artificial Intelligence, 2020.
Soman, K., Rose, P. W., Morris, J. H., Akbas, R. E., Smith, B., Peetoom, B., Villouta-Reyes, C., Cerono, G., Shi, Y., Rizk-Jackson, A., Israni, S., Nelson, C. A., Huang, S., and Baranzini, S. Biomedical knowledge graph-optimized prompt generation for large language models. Bioinfor- matics, 40, 2023.
Xu, W., Chen, K., and Zhao, T. Discriminative reasoning for document-level relation extraction. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 1653- 1663, 2021b.
Sun, Q. Huang, K., Yang, X., Tong, R., Zhang, K., and Poria, S. Consistency guided knowledge retrieval and denoising in Ilms for zero-shot document-level relation triplet extraction. Proceedings of the ACM on Web Con- ference 2024, 2024.
Xu, X., Cao, Y., Hu, H., Xiang, H., Qi. L.. Xiong, J., and Dou, W. MGF-ESE: An enhanced semantic extractor with multi-granularitg feature fusion for code summarization. In THE WEB CONFERENCE 2025, 2025a.
Xu, X., Zhou, Y., Xiang, H., Li, X., Zhang, X., Qi, L., and Dou, W. Nlgt: Neighborhood graph transformer framework for node classification. Pro- ceedings of the AAAI Conference on Artificial Intelli- gence, 2025b.
Tan, Q. He, R., Bing. L., and Ng. H. T. Document-level re- lation extraction with adaptive focal loss and knowledge distillation. In Findings of the Association for Computa- tional Linguistics: ACL 2022, pp. 1672-1681, 2022a.
and label-enhanced
Xue, L., Zhang, D., Dong, Y., and Tang. J. AutoRE: Document-level relation extraction with large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pp. 211 211-220, 2024.
Tan, Q. Xu, L., Bing, L., Ng. H. T., and Aljunied, S. M. Revisiting DocRED- addressing the false negative prob- lem in relation extraction. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 8472 8487, 2022b.
Yao, Y., Ye, D., Li, P., Han, X., Lin, Y., Liu, Z., Liu, Z., Huang. L., Zhou, J., and Sun, M. DocRED: A large-scale document-level relation extraction dataset. In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 764-777, 2019.
Tang, H., Cao, Y., Zhang. Z., Cao. J.. Fang. F., Wang, S., and Yin, P. Hin: Hierarchical inference network for document- level relation extraction. Advances in Knowledge Discov- ery and Data Mining, 12084:197 -009, 2020a.
Tang, Y., Huang, J., Wang, G., He, X., and Zhou, B. Orthog- onal relation transforms with graph context modeling for knowledge graph embedding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 27122 2020b.
Zeng, S., Xu, R., Chang, B., and Li, L. Double graph based reasoning for document-level relation extraction. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1630-1640, 2020.

---

DocKS-RAG: Optimizing Decument-Level Relation Extraction through Hybrid Prompt Tuning
Zeng, S., Wu, Y., and Chang, B. SIRE: Separate intra- and inter-sentential reasoning for document-level relation extraction. In Findings of the Association for Compu- tational Linguistics: ACL-IJCNLP 2021, pp. 524-534, 2021.
Zhang, N., Chen, X., Xie, X., Deng, S., Tan, C., Chen, M., Huang, F., Si, L., and Chen, H. Document-level rela- tion extraction as semantic segmentation. In Proceedings of the Thirtieth International Joint Conference on Artifi- cial Intelligence, IJCAI 2021, Virtual Event Montreal, Canada, 19-27 August 2021, pp. 3999-4006, 2021.
Zhou, W., Huang, K., Ma, T., and Huang, J. Document-level relation extraction with adaptive thresholding and local- ized context pooling. In AAAI Conference on Artificial Intelligence, 2020.
12

---

DocKS-RAG: Optimizing Decument-Level Relation Extraction through LLM-Enhanced Hybrid Prompt Tuning
A. Appendix
SIRE (Zeng et al., 2021) mitigates the negative influ- ence caused by indiscriminate representation by differ- entiating the representation of relations and implement- ing a comprehensive logical reasoning module.
A.1. Datasets
We evaluate our proposed DocKS-RAs RAGG on two widely used benchmark datasets in document-level RE, which are shown follows:
SSAN (Xu et al., 2021a) proposes two alternative trans- formation modules to produce attentive biases for adap- tively regularization.
DocRED (Yao al., 2019) is common benchmark for the document-level RE tasks, comprising diverse set of documents extracted from Wikipedia. The goal of this dataset is to extract the structured triplets from documents and it allows for the evaluation of models' abilities to identify relations by pross-sentence, cross- paragraph and even cross-discourse, which reflects the complexities inherent in real-world documents.
DocuNet (Zhang et al., 2021) introduces an entity-level relation matrix and incorporates both context infor- mation and global dependencies through a U-shaped segmentation module.
KD-DocRE (Tan et al., 2022b) adopts knowledge distillation method to deal with the discrimination be- tween the annotated texts and distantly supervised data.
Re-DocRED (Tan al., 2022a) is an advanced exten- sion of the original DocRED dataset. It retains the rich and diverse annotations of entity pairs and their rela- tions from its predecessor DocRED, and introduces relational reasoning aspect that enables models to bet- ter capture the complex interactions and dependencies between entities and relations from extensive documen- tation data.
KMGRE (Jiang et al., 2022) presents a mention-level relation extractor to solve the multi-label problem in optimizing the mention-level relation extractor.
AutoRE (Xue et al., 2024) presents innovative rela- tional extraction paradigm for LLMs that obviates the dependence on pre-defined relations and emphasizes the identification of triplet facts.
A.2. Baselines
LMRC (Li et al., 2024) addresses the performance gap in between LLMs and traditional approaches by proposing novel classifier-LLMM methodology that effectively directs the LLM's attention towards relevant entity pairs.
We evaluate proposed framework by comparing with recently presented PLMs-based, graph-based based and LLMs- based methods, which are listed as follows:
HIN (Tang et al., 2020a) tackles the challenge in multi- granularity inference by leveraging different levels of contextual information for accurate extraction.
A.3. Implementation Details
The technical modules of DocKS-RAG primarily consist of document-Level knowledge graph construction, sentence- level knowledge base construction, and sybriddprompp tun- ing. The specific experimental parameters are set as follows:
ATLOP-BERS (Zhou et al., 2020) obtains a more com- prehensive representation of the entities by transferring attention from the PLMs to capturing the associated contexts.
SRLR (Huang et al., 2023) tackles the challenges in the relation representation and logical reasoning by extending the extraction to the mention level.
Document-Level Knowledge Graph Construction and Retrieval. We employ general graph neural net- work for the construction of Document-level Knowl- edge graph (DocKG) and later extract structural knowl- edge from DocKG by graph retrieval for further hybrid prompt generation. In the stage of construction, both the entity and relation embedding dimensions are 128, and the network initialized with convolutional lay- ers, which aggregate the neighborhood information and capture the long-range dependencies within the document. Dropout is set rate of 0.1 in the con- volutional layers to mitigate the risk of overfitting. To enhance the stability of training, Adam is employed as the optimizer. Learning rate is set to le-3, and weight decay is 5c-4. In the stage of retrieval, we establish a minimum threshold at 0.7 for measuring the similarity,
GLRE (Wang al., 2020) proposes method to code document information through global and local representation of entities and aggregate with the con- text information for better extraction.
LSR (Nan et al., 2020) particularly considers capturing the complex interactions between inter-sentence enti- ties and employing a refinement strategy for multi-hop reasoning.
HeterGSAN (Xu et al., 2020) proposes the reconstruc- tion of ground-trum -ouut path dependencies and focuses on the relevant entity pairs improve the accuracy of RE.
13

---

DocKS-RAG: Optimizing Decument-Level Relation Extraction through LLMM-nhanced Hybrid Prompt Tuning
which guarantees that only highly relevant subgraphs will be extracted. We transfer the relevant subgraphs to informative prompts with a maximum length of 128 tokens.
Knowledge Base Construction and Semantic Retrieval. We harness BGE embedding model for sentence representation. In the semantic retrieval component, Top5 relevant sentences will be retrieved associated with the user query, which are further transferred into informative prompts with maxi- mum length 512 tokens.
Sentence-Level
Hybrid-Prompt Tuning. We adopt LLaMA3-8B as the backbone LLM. LORA is chosen as the adapter training method to fine-tune LLaMA3-8B. The low- rank setting the adapter is 16, along with the warmup ratio of 0.1. We the learning rate 5e-5, and the batch size is 2 for both the training and evaluation stage. We utilize learning rate scheduler with cosine variations for training, and we conduct fine-tuning for 3 epochs. All experiments conducted in this paper are implemented by Py PoToch and are trained on four 24GB RTX 3090 GPUs.
14


Reviews:
[{"review_id": "BGK23sBu6u", "reviewer": "Reviewer_CMNB", "summary": "The paper introduces DocKS-RAG, a novel framework aimed at enhancing document-level relation extraction (RE) by integrating large language models (LLMs) with structured knowledge graphs. The proposed method combines a Document-level Knowledge Graph (DocKG) with a Sentence-level Semantic Retrieval-Augmented Generation (SetRAG) mechanism to improve entity-relation understanding in complex documents. The authors conduct extensive experiments on benchmark datasets, DocRED and Re-DocRED, demonstrating that DocKS-RAG significantly outperforms existing PLM-based and graph-based methods by achieving superior F1 and Ign-F1 scores, thereby validating its effectiveness in addressing the intricacies of document-level RE tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive comments and insightful questions.\n\nQ1: While DocKS-RAG achieves high performance, the ablation studies suggest that even without blending structural and semantic components, competitive results can be obtained. For example, simpler setups without DocKG or SetRAG achieve reasonably good scores. Could the authors elaborate on the trade-offs between complexity and efficiency in practical deployments?\n\nResponse to Q1: We appreciate your insightful question. In practical applications, we take full advantage of the parallel processing capabilities provided by both DocKG and SetRAG, enabling us to handle tasks efficiently while maintaining high performance of relational extraction. Additionally, by applying threshold retrieval strategies, we could further simplify the complexity of our operations.\n\nQ2: In your discussion, you mentioned that existing graph-based methods can introduce noise and irrelevant connections, affecting performance. Can you elaborate on how DocKS-RAG specifically mitigates this noise during knowledge graph construction and retrieval processes?\n\nResponse to Q2: Thank you for your valuable feedback. As presented in Section 4, firstly, DocKS-RAG applies a threshold parameter $ \\tau_{er} $ during graph retrieval to filter out extraneous entities. Secondly, another threshold parameter $ \\tau_{eq} $ is used to retrieve the relevant contextual sentences. Finally, DocKS-RAG generates hybrid prompts that combine the information from both the DocKG and SetRAG, which integrates relevant contextual information to enhance semantic alignment, improving the overall extraction performance.\n\nQ3: Regarding future work, how do you envision scaling DocKS-RAG to accommodate different types of documents or domains with vastly different structure or language use (e.g., legal texts, scientific literature)? What modifications might be necessary to adapt the framework effectively?\n\nResponse to Q3: To scale DocKS-RAG for diverse document types or domains, modifications could include tailoring the DocKG construction to account for domain-specific terminology and relationships, alongside enhancing the SetRAG mechanism to accommodate varied syntactical structures and language use. Additionally, training additional domain-specific models or fine-tuning existing ones with targeted datasets would improve adaptability and semantic accuracy for specific contexts.", "claims_and_evidence": "The claims made in the submission regarding the efficacy and advantages of the DocKS-RAG framework in document-level relation extraction (RE) appear to be generally supported by clear and convincing evidence. The authors substantiate their claims through extensive experimental results, comparing their framework against existing state-of-the-art methods on benchmark datasets like DocRED and Re-DocRED.", "methods_and_evaluation_criteria": "Conducting ablation studies to evaluate the contribution of different components validates the efficacy of each aspect of the framework. This methodological rigor strengthens the argument for the importance of integrating both structural and contextual information in improving extraction performance.", "theoretical_claims": "This paper primarily focuses on empirical methodologies rather than formal theoretical proofs. However, it does include theoretical claims regarding the effectiveness of the proposed methods, particularly the integration of structural knowledge and contextual understanding through the DocKS-RAG framework.", "experimental_designs_or_analyses": "The authors compare DocKS-RAG against several established methods, including both PLMs-based and graph-based approaches. This comparative analysis is essential for establishing the framework's effectiveness.", "relation_to_broader_scientific_literature": "Prior to this study, substantial efforts were made to enhance relation extraction primarily through sentence-level methods using pre-trained language models (PLMs), such as BERT (Devlin et al., 2019) and REBEL (Huguet Cabot & Navigli, 2021). The present paper's introduction of a document-level approach, specifically through the DocKS-RAG framework, builds on this prior work by addressing the limitations of existing PLMs-based methods, thus contributing to ongoing dialogues about enhancing RE capabilities.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths:  \n- The paper addresses a critical gap in natural language processing by focusing on document-level relation extraction, which is vital for tasks like knowledge graph construction and information retrieval. \n- Extensive experiments on DocRED and Re-DocRED benchmarks are robust, showcasing superior performance in both F1 and Ign-F1 scores compared to state-of-the-art methods.\n- Comprehensive ablation studies clarify the contributions of individual components (DocKG, SetRAG, and hybrid-prompt tuning), reinforcing the validity of the design choices. \n\nWeakness:    \n- While the authors highlight the novelty of combining graph-based and LLM-based methods, the range of comparison baselines is limited. Existing works on the BioRED [1] dataset and similar benchmarks in knowledge extraction provide a richer set of methods that could offer additional insights into the framework’s relative performance. Incorporating these comparisons could strengthen the claim of novelty and demonstrate broader applicability. \n- While DocKS-RAG achieves high performance, the ablation studies suggest that even without blending structural and semantic components, competitive results can be obtained. For example, simpler setups without DocKG or SetRAG achieve reasonably good scores. Could the authors elaborate on the trade-offs between complexity and efficiency in practical deployments?\n\nReference:  \n[1] Islamaj, Rezarta, et al. \"The overview of the BioRED (Biomedical Relation Extraction Dataset) track at BioCreative VIII.\" Database 2024 (2024): baae069.", "other_comments_or_suggestions": "Since the paper targets a broader audience, adding a small section that explains potential practical applications and user scenarios for the proposed framework could make the contributions more relatable. People appreciate understanding how theoretical advancements may impact real-world applications.", "questions_for_authors": "- In your discussion, you mentioned that existing graph-based methods can introduce noise and irrelevant connections, affecting performance. Can you elaborate on how DocKS-RAG specifically mitigates this noise during knowledge graph construction and retrieval processes?\n- Regarding future work, how do you envision scaling DocKS-RAG to accommodate different types of documents or domains with vastly different structure or language use (e.g., legal texts, scientific literature)? What modifications might be necessary to adapt the framework effectively?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VfVJmsm7ix", "reviewer": "Reviewer_Amvq", "summary": "In this work, the authors introduce DocKS-RAG, a framework that enhances large language models for document-level relation extraction. By integrating structural knowledge from a Document-level Knowledge Graph (DocKG) with semantic insights from a Sentence-level Semantic Retrieval-Augmented Generation (SetRAG) mechanism, the framework effectively captures complex relationships in documents. The paper emphasizes the importance of aligning structural and semantic knowledge to address the noise associated with traditional methods. Experiments on DocRED and Re-DocRED demonstrate that DocKS-RAG significantly improves accuracy and highlights the advantages of hybrid-prompt tuning techniques.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback on our paper.\n\nC1: When mentioning the sentence-level Semantic Retrieval-Augmented Generation, it might be useful to briefly explain how it differs from typical retrieval mechanisms.\n\nResponse to C1: Thank you for your valuable comment. Our proposed SetRAG module constructs a knowledge base from segmented sentences and retrieves contextually relevant information based on embeddings, allowing for improved understanding of the document's semantics. Such semantic information will be integrated with structural knowledge from DocKG, which further enhances the performance of LLMs in relation extraction tasks.\n\nQ1: Although DocKS-RAG demonstrates high performance, the ablation studies indicate that competitive results can still be achieved without integrating structural and semantic components. For instance, simpler configurations that do not utilize DocKG or SetRAG yield reasonably good scores. Could the authors elaborate on the trade-offs between complexity and efficiency in practical deployments?\n\nResponse to Q1: Thank you for your thoughtful question. As shown in Table 4, the experimental results indicate that the integration of the DocKG and the SetRAG mechanisms significantly enhances model performance, demonstrating the effectiveness of our proposed DocKS-RAG module. In practical deployment scenarios, we capitalize on the parallelization capabilities offered by both DocKG and SetRAG, which allows for efficient processing without compromising the quality of relational extraction. Furthermore, by implementing strategies such as threshold retrieval, we manage to streamline the complexity of the operations involved. Integrating both modules enhances the model's adaptability to various document types and ensures robust performance in real-world applications.", "claims_and_evidence": "The authors claim that their proposed framework, DocKS-RAG, addresses limitations in existing document-level relation extraction approaches by effectively combining linguistic and structural knowledge. Extensive experiments demonstrate that DocKS-RAG achieves superior performance metrics compared to state-of-the-art methods, as evidenced by significant gains in F1 and Ign-F1 scores. Additionally, the systematic ablation studies offered in the paper highlight the significance of each component, fostering confidence in the robustness of the framework's design and functionality.", "methods_and_evaluation_criteria": "In this paper, the authors use a Document-level Knowledge Graph (DocKG) alongside a Sentence-level Semantic Retrieval-Augmented Generation (SetRAG) mechanism, which allows the framework to capture both the structural relationships among entities and the contextual semantics of the text. This dual approach addresses the semantic misalignment typically observed between pre-trained language models and knowledge graphs, which is particularly crucial for complex document-level tasks.", "theoretical_claims": "The paper asserts that employing a hybrid-prompt tuning approach, coupled with Parameter-Efficient Fine-Tuning (PEFT), leads to improved adaptability and performance of LLMs. This claim remains largely substantiated by experimentation.", "experimental_designs_or_analyses": "An ablation study is conducted by removing different components of the DocKS-RAG framework, showing the impact of each component on performance. However, the nuances of the results could be better articulated—explaining why the removal of specific components leads to performance degradation would provide deeper insights into the framework's mechanics.", "relation_to_broader_scientific_literature": "The study's emphasis on integrating structural knowledge via Document-level Knowledge Graphs (DocKG) echoes findings in the literature that suggest the synergy between PLMs and knowledge graphs (KGs) can improve contextual understanding. The approach taken in this paper thus not only aligns with but also advances previous investigations into the necessity of structural information in processing complex relationships within texts.", "essential_references_not_discussed": "The references are enough.", "other_strengths_and_weaknesses": "The idea of combining multiple retrieval and KG creation methods to form a RAG-based approach with heterogeneous data is interesting. The main novelty stems from the DocKG approach. The DocKG is generated by predicting if a relation should be added between two entities, given the embedding of the entities and the relation. The authors should ensure that all acronyms such as PLMs and KGs have been defined the first time they appear. This aids readers who may be unfamiliar with these terms.", "other_comments_or_suggestions": "When mentioning the sentence-level Semantic Retrieval-Augmented Generation, it might be useful to briefly explain how it differs from typical retrieval mechanisms.", "questions_for_authors": "Although DocKS-RAG demonstrates high performance, the ablation studies indicate that competitive results can still be achieved without integrating structural and semantic components. For instance, simpler configurations that do not utilize DocKG or SetRAG yield reasonably good scores. Could the authors elaborate on the trade-offs between complexity and efficiency in practical deployments?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2FWgqi0exs", "reviewer": "Reviewer_V77s", "summary": "In this paper, the authors propose a DocKS-RAG method to combine structural knowledge and semantic information for document-level relation extraction task. In DocKS-RAG, the authors first rely on GNNs to construct a document-Level knowledge graph and retrieve relevant information from this graph according to the user query. Then, they extract relevant sentences from the input document. On this basis, they construct hybrid prompts and adopt LORA to train an LLM. Extensive experiments on two datasets verify the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for recognizing and affirming our work. Regarding the questions raised during the review process, we have carefully considered them and provided detailed responses as follows:\n\nQ1: In Introduction, the authors state that “graph-based methods … lack of sufficient contextual information”. The authors need to explain why they believe graph-based methods lack contextual information and what specific types of context are missing.\n\nResponse to Q1: Thank you for your valuable question. Graph-based methods primarily focus on the structural relationships between entities, which can obscure essential semantic nuances due to the static nature of graph representations. In complex document-level relation extraction tasks, entities may relate across texts, paragraphs, or even chapters within a document. This limitation in the graph-based approach restricts our understanding of how entities interact in different contexts, ultimately hindering the effectiveness of extraction.\n\nQ2: Also in Introduction, the authors state that “PLMs are … the semantic misalignment between the texts and graphs”. I don’t understand how they address this issue in this paper.\n\nResponse to Q2: As shown in Section 4.4, we integrate informative structural knowledge from DocKG with the relevant sentences retrieved by SetRAG, and further generate the hybrid prompts to enhance the interaction between the textual data and the graph-based representations, thereby improving relation extraction performance.\n\nQ3: Where do the training samples come from? Besides, will this process limit the applicability of the method to other domains?\n\nResponse to Q3: Thank you for your insightful questions. The training samples for constructing DocKG come from observable documentation data, which is utilized to obtain representations of entities and relations. In our proposed DocKS-RAG framework, the training process entails the construction of DocKG and SetKB. Relevant subgraphs and sentences are retrieved to create hybrid prompts, which are then fine-tuned using PEFT to improve the performance of LLMs in document-level relation extraction tasks. Although DocKS-RAG relies on this training process, it is adaptable and can be applied to other domains by fine-tuning the model with domain-specific data.\n\nQ4: In Eq.(10), the authors introduce a generation function $ g_{DocKG} $. How do they implement this function?\n\nResponse to Q4: Thank you for your thoughtful insights. $ g_{DocKG} $ is implemented as a function for graph knowledge transformation. We pre-defined various entity-relation mapping types. For example, if a subgraph has \"Wisconsin\" and \"U.S.\" connected by the edge \"state\", it would be transformed into the statement \"Wisconsin is a state of U.S.\" based on our mappings. Ultimately, we concatenate such statements with the relevant semantic information retrieved from SetRAG, and obtain the hybrid prompts.\n\nC1: When explaining the extraction of knowledge graphs from the knowledge base, the authors mention the user query q. However, in Section 3 (Problem Definition), there is no indication that a user query is provided, and in Figure 1, the query is also not visible. The authors need to clarify this inconsistency.\n\nResponse to C1: As mentioned in  Section 4, q is denoted as the input user query for further extraction. We appreciate your insight and will consider including the explanations about q in the Problem Definition section and Figure 1 in the revised manuscript to enhance the readability.\n\nC2: In Section 5, the authors provide the hyper-parameter experiments before the ablation study, which seems to be weird for me. I suggest rearranging the order to improve clarity and coherence.\n\nResponse to C2: Thank you for your valuable suggestion. We will consider making this change to the revised version.", "claims_and_evidence": "The experiments on two widely-used benchmarks demonstrate the effectiveness of the method. Additionally, the ablation study shows the importance of each component in the method.", "methods_and_evaluation_criteria": "I believe that the proposed method is suitable for the problem. The motivation for combing structural knowledge and semantic information makes sense and is very important. As for knowledge construction, the proposed GNN-based method adopts a general pipeline in existing literature. As for semantic information, the extraction process is also acceptable. Additionally, the benchmarks and metrics are reasonable.", "theoretical_claims": "This paper does not make any theoretical claims.", "experimental_designs_or_analyses": "Yes, I have checked the details in Section 5, including the experimental setup and result analyses. The experiments in Section 5.5 verify the overall effectiveness and the influence of each hyper-parameter, while the results in Section 5.6 verify the importance of each module in the proposed method.", "relation_to_broader_scientific_literature": "In my opinion, the proposed method provides a general perspective to address the issues of document-level RE task. The idea of integrating structural knowledge and semantical information is also general enough to extend to our domains or tasks.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "The main strengths of this paper include:\n\n1.\tThe motivation of combining structural knowledge and semantic information makes sense to me and is important.\n\n2.\tThe proposed method can effectively achieve the goal in this paper.\n\n3.\tThe experimental comparison with 14 baselines (including some SOTA methods) is solid and convincing.\n\nThe main weakness of this paper include:\n\n1.\tMany technical details need to be further illustrated (please see the questions 3-5 below).\n\n2.\tThe technical contributions of this paper seem to be limited. The methods used to extract knowledge and relevant sentences in the document are very common in the literature. I don’t find essential improvements of the method.\n\n3.\tThe experimental section needs to be further revised (please see the question 6 below).", "other_comments_or_suggestions": "There are some typos in this paper, such as:\n\n1.\tLine 120, “model” should be “models”\n\n2.\tLine 351, “a extra” should be “an extra”", "questions_for_authors": "1.\tIn Introduction, the authors state that “graph-based methods … lack of sufficient contextual information”. The authors need to explain why they believe graph-based methods lack contextual information and what specific types of context are missing.\n\n2.\tAlso in Introduction, the authors state that “PLMs are … the semantic misalignment between the texts and graphs”. I don’t understand how they address this issue in this paper.\n\n3.\tIn Section 4.2, the proposed method needs to train entity pair representations to construct the document-level knowledge graph. Where do the training samples come from? Besides, will this process limit the applicability of the method to other domains?\n\n4.\tIn Eq.(10), the authors introduce a generation function g_{DocKG}. How do they implement this function?\n\n5.\tWhen explaining the extraction of knowledge graphs from the knowledge base, the authors mention the user query q. However, in Section 3 (Problem Definition), there is no indication that a user query is provided, and in Figure 1, the query is also not visible. The authors need to clarify this inconsistency.\n\n6.\tIn Section 5, the authors provide the hyper-parameter experiments before the ablation study, which seems to be weird for me. I suggest rearranging the order to improve clarity and coherence", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iTFhGEGJnT", "reviewer": "Reviewer_RKaC", "summary": "The authors of the paper propose a novel approach for document-level relation extraction. During the training phase, they prepare two additional texts: one sourced from DocKG and the other from SetRAG, which are concatenated and utilized as a prefix in the final prompt. Subsequently, they fine-tune a small open-source language model to predict the relations present from the input document.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback.\n\nC1: Although the paper provides a detailed methodology, some sections could benefit from greater clarity. For instance, the hybrid prompt generation process may require more concrete examples to fully illustrate the differences and advantages over conventional methods.\n\nResponse to C1: We appreciate your feedback, particularly regarding the clarity of the hybrid prompt generation process. We aim to enrich this section by including specific examples that illustrate how the generated prompts differ from those produced by conventional methods. By showcasing practical scenarios, we intend to clarify the advantages of our hybrid approach and its effectiveness in providing richer context for relation extraction tasks.\n\nQ1: Could you provide a detailed explanation of the operations of Formula 10 and Formula 15?\n\nResponse to Q1: Thank you for your question. As illustrated in Section 4, the purpose of Formula 10 is to generate informative prompts based on the retrieved entities and relations from DocKG. It defines the prompt generation function $ g_{DocKG} $, which takes a set of relevant entities and relations as input and outputs structured prompts $ P_{DocKG} $​.  Formula 15 constructs hybrid prompts using the retrieved sentences from SetKB. It employs the generation function $ g_{SetRAG} $​ to create prompts from the relevant triplets $ T_k $​ while appending the retrieved sentences $ S_k $​. This integration enriches the prompts by combining contextual insights with structured knowledge, enhancing the model's ability to perform document-level relation extraction accurately.\n\nQ2: In your results, you mention challenges related to semantic misalignment between PLMs and knowledge graphs. Can you elaborate on how your framework specifically addresses this issue?\n\nResponse to Q2: Thank you for your thoughtful insights. As mentioned in Section 4.2, the embedding spaces of PLM and DocKG differ significantly. To address this issue, we integrate PLM embeddings as the initial inputs for entities and relations, which allows DocKG to learn from the PLM during the updating process and improves the reliability of similarity calculations.\n\nQ3: Could you clarify the process for generating these hybrid prompts, and perhaps provide examples of how they differ from standard prompts?\n\nResponse to Q3: Thank you for your insightful feedback. As shown in Section 4.4, we incorporate both relevant structural and semantic information from DocKG and SetRAG. Specifically, we pre-defined various entity-relation mapping types. For example, if a subgraph has \"Wisconsin\" and \"U.S.\" connected by the edge \"state\", it would be transformed into the statement \"Wisconsin is a state of U.S.\" based on our mappings. Ultimately, we concatenate such statements with the relevant semantic information retrieved from SetRAG, and obtain the hybrid prompts.", "claims_and_evidence": "Yes. The experiments conducted are on specific benchmark datasets; hence, the generalizability of the results to other domains or types of documents remains somewhat unspecified.", "methods_and_evaluation_criteria": "Yes. The selection of benchmark datasets specific to document-level relation extraction is appropriate. Both datasets are well-established in the field and are designed to evaluate models on complex, multi-entity document scenarios.", "theoretical_claims": "Yes. The choice of using F1 and Ign-F1 scores as evaluation metrics is justified within the context of document-level RE tasks. The theoretical underpinning of why these metrics are particularly suitable for assessing the model's capabilities in capturing implicit relations could be discussed more rigorously.", "experimental_designs_or_analyses": "Yes, the experiment design and analysis is suitable.", "relation_to_broader_scientific_literature": "Unlike traditional methods that often rely heavily on fine-tuning PLMs, the paper innovatively proposes a hybrid approach that enhances model adaptability by combining structural knowledge with contextual prompts, thus reflecting a significant evolution from earlier works that typically did not fully harness the potential of LLMs in this context.", "essential_references_not_discussed": "NA.", "other_strengths_and_weaknesses": "Strengths:\n1)The paper is well-organized, with a clear delineation of the problem, proposed methodology, and experimental results. \n2)The authors present their ideas logically, making it accessible for readers who may not have extensive background knowledge in document-level relation extraction.\n\nWeaknesses:\n1)The hybrid prompt generation process is complex and may be difficult for users unfamiliar with the underlying methodologies to interpret or adjust, which could limit its accessibility.\n2)The emphasis on document-level relation extraction may not fully address the challenges posed by unstructured, noisy, or overly complex input data.", "other_comments_or_suggestions": "1)For further understandability, it would also makes sense to put some numbers in Figure 2 for each step e.g. GNN training which are then repeated in the corresponding section to guide the reader a bit more. \n2)Although the paper provides a detailed methodology, some sections could benefit from greater clarity. For instance, the hybrid prompt generation process may require more concrete examples to fully illustrate the differences and advantages over conventional methods.\n3)Could you provide a detailed explanation of the operations of Formula 10 and Formula 15?", "questions_for_authors": "1)In your results, you mention challenges related to semantic misalignment between PLMs and knowledge graphs. Can you elaborate on how your framework specifically addresses this issue?\n2)The authors propose a hybrid prompts generation method that combines structural interactions with semantic information. Could you clarify the process for generating these hybrid prompts, and perhaps provide examples of how they differ from standard prompts?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents DocKS-RAG, a novel framework that enhances document-level relation extraction through the integration of structural knowledge from a Document-level Knowledge Graph (DocKG) with semantic information from a Sentence-level Semantic Retrieval-Augmented Generation (SetRAG) mechanism. It effectively addresses limitations in current approaches by combining linguistic and structural knowledge in an innovative way.

I recommend accepting this paper as three reviewers all gave positive comments. The integration of structural knowledge with contextual understanding is both innovative and well-justified. The experimental results are consistently good across established benchmarks, with comprehensive ablation studies validating the contribution of each component.
Decision: Accept (poster)