AutoRE: Document-Level Relation Extraction with Large Language Models (2024)

Abstract: Large Language Models (LLMs) have demonstrated exceptional abilities in comprehending and generating text, motivating numerous researchers to utilize them for Information Extraction (IE) purposes, including Relation Extraction (RE). Nonetheless, most existing methods are predominantly designed for Sentencelevel Relation Extraction (SentRE) tasks, which typically encompass a restricted set of relations and triplet facts within a single sentence. Furthermore, certain approaches resort to treating relations as candidate choices integrated into prompt templates, leading to inefficient processing and suboptimal performance when tackling Document-Level Relation Extraction (DocRE) tasks, which entail handling multiple relations and triplet facts distributed across a given document, posing distinct challenges. To overcome these limitations, we introduce AutoRE, an endto-end DocRE model that adopts a novel RE extraction paradigm named RHF (Relation-HeadFacts). Unlike existing approaches, AutoRE does not rely on the assumption of known relation options, making it more reflective of realworld scenarios. Additionally, we have developed an easily extensible RE framework using a Parameters Efficient Fine Tuning (PEFT) algorithm (QLoRA). Our experiments on the RE-DocRED dataset showcase AutoRE’s best performance, achieving state-of-the-art results, surpassing TAG by 10.03% and 9.03% respectively on the dev and test set. The code is available1 and the demonstration video is provided2.

heading: 1 Introduction
text: The rise of LLMs, such as GPT-4 (Achiam et al., 2023) and Llama2 (Touvron et al., 2023), has significantly propelled the progress of natural language processing due to their strong capabilities
1https://github.com/THUDM/AutoRE 2https://www.youtube.com/watch?v=IhKRsZUAxKk ⋆LX and DZ contributed equally. †JT is the corresponding author.
in text understanding, generation, scientific reasoning (Zhang et al., 2024a,b), social bot detection (Zhou et al., 2024), and generalization (Zhao et al., 2023). There has been an increasing interest in using LLMs to generate structured information for IE tasks (Xu et al., 2023; Wadhwa et al., 2023) and making impressive progress. Typical IE tasks using LLMs include Named Entity Recognition (NER) (Wang et al., 2023a), Relation Extraction (RE) (Zhou et al., 2023), and Event Extraction (EE) (Xu et al., 2023). Despite the outstanding result, the performance of current LLMs in RE is still far from satisfactory.
Underperformance on DocRE Tasks. We evaluated several high-performing LLMs on the document-level RE (DocRE) task, specifically using the test set of Re-DocRED (Tan et al., 2022). These models included GPT-3.5turbo3(ChatGPT), Mistral-7B-Instruct-v0.2 (Jiang et al., 2023) (Mistral-7B), Vicuna-7B-v1.5 (Chiang et al., 2023) (Vicuna-7B), and ChatGLM3-6B (Du et al., 2022). Our results indicate that, without specific fine-tuning, the performance of these language
3https://chat.openai.com/chat
211
models on DocRE tasks is suboptimal, as shown in the blue bars in Figure 1.
Inefficacy in Multi-Relations. Incorporating relations directly into the prompt template as candidates is a common strategy for LLM-based models (Wang et al., 2023b; Wei et al., 2023; Xiao et al., 2023). This method is effective for tasks that involve a relatively small number of relation types. However, the number of relation types can easily exceed 100 in real-world scenarios. Dealing with multiple relations, as seen in the Re-DocRED dataset with its 96 relation types, presents a significant challenge for most existing models. Embedding such many relations directly into the prompt template is often impractical (Wadhwa et al., 2023).
Limitations of Current RE Paradigms. The current paradigms in RE exhibit significant limitations in their effectiveness. Modern generative methods typically operate by either directly producing triplet facts from the input text in a singular step (Wang et al., 2023b), or by initially identifying a set of relations and subsequently generating triplet facts based on these relations (Wei et al., 2023). Earlier approaches prioritized the extraction of the head entity before the derivation of triplet facts (Li et al., 2019). However, these methodologies fall short of handling DocRE tasks that involve multiple relations and plenty of triplet facts. For instance, a single instance in the Re-DocRED dataset might encompass as many as 27 different relations and include up to 142 distinct triplet facts.
To address challenges identified in existing RE paradigms, we innovate a new pipeline RE paradigm, Relation-Head-Facts (RHF). We comprehensively redefined the 96 relation descriptions and crafted simplified relation extraction templates, developing an instruction-tuning dataset based on Re-DocRED. Utilizing the Mistral-7B model with Parameter Efficient Fine-Tuning (PEFT), QLoRA (Dettmers et al., 2023), our model achieved stateof-the-art (SOTA) performance on the Re-DocRED test set. Key contributions of our work include:
Various RE Paradigms. We conducted experiments across a variety of RE paradigms and revealed that a pipeline RE approach is especially potent for DocRE, particularly RHF. This paradigm prioritizes the extraction of relations, followed by the identification of subjects, thereby significantly enhancing the model’s capacity to efficiently and accurately uncover triplet facts.
Efficient DocRE Model. Adopting the RHF paradigm for DocRE and refined relation descriptions, we have meticulously crafted an instructionfinetuning dataset based on Re-DocRED. This dataset was utilized to fine-tune the Mistral-7B with QLoRA, culminating in the creation of AutoRE, which achieved SOTA results across multiple pre-trained LLMs (PLMs), demonstrating the generality and effectiveness of this model architecture.
Easy Enhancement of Capabilities. We have incorporates three distinct QLoRA modules within the RHF framework, where each module is exclusively responsible for a specific task: one for relation extraction, another for head entity identification, and the third for triple fact extraction, ensuring specialized handling for each aspect. This strategy effectively lays the groundwork for future advancements while ensuring a minimal rise in computational demands and avoiding interference between subtasks.

heading: 2 Related work
text: DocRE refers to the task of extracting relations between entities at the document level, we follow the definition in (Zheng et al., 2023): Given a document D with a set of sentences containing a set of entities V = {ei}Ni=1. The DocRE task is to predict the relation types between an entity pair (eh, et)h,t∈{1,··· ,N},h̸=t, where h stands for the head (subject) and t stands for the tail (object).
LLMs for DocRE. Researchers have been employing LLMs to tackle RE tasks. For example, ChatIE (Wei et al., 2023) deconstructs the complex RE process into assembling the outputs from multiple rounds of Question-Answer into a final structured format. PromptRE (Gao et al., 2023) integrates LLM prompting with data programming to deal with DocRE. However, the performance of LLMs on RE tasks still lags behind SOTA models. Han et al. concluded that ChatGPT does not adequately comprehend the subject-object relationships in RE tasks. Similarly, Li et al. noted that in Standard-IE settings, ChatGPT’s performance is generally not as effective as BERT-based models. Moreover, most models are tested on SentRE. To test LLMs for DocRE, we conducted tests on ChatGPT, Mistral-7B, Vicuna-7B, and ChatGLM3-6B and revealed that the current performance is far from satisfactory, as illustrated in Figure 1. This aligns with findings reported by (Li et al., 2023b),
indicating that current models still fall significantly short in performance on DocRE.
RE Prompt Template. These models finetuned on LLMs for RE operate on a prompt-based or instruction-driven mechanism (Beurer-Kellner et al., 2023), engaging in a question-and-answer format to execute RE tasks. ChatIE (Wei et al., 2023), InstructUIE (Wang et al., 2023b), and YAYI (Xiao et al., 2023) while demonstrating formidable capabilities in IE, exhibit considerable limitations in their RE prompt templates. A common method in their RE process involves embedding a list of relations into the model’s prompt template as alternatives. However, this approach becomes impractical when dealing with DocRE, such as the 96 relations in the Re-DocRED. This limitation is acknowledged by Wadhwa et al., who concludes that “for datasets with long texts or a large number of targets, it is not possible to fit detailed instructions in the prompt”.
RE Paradigms. Within the context of LLMs, RE paradigms are primarily categorized into two types: Pipeline and Joint. The Pipeline approach involves first identifying relations and then extracting triplet facts, or initially extracting a head entity followed by its corresponding relation and tail entity. This approach deviates from the traditional methodology of first extracting entities and then determining their interrelations (Chen and Guo, 2022; Jiang et al., 2020). The main drawbacks is that applying the conventional Pipeline approach to LLMs can be extremely time-consuming, particularly when many entities lack interrelations. On the other hand, the Joint paradigm, which inputs a text and directly outputs all triplet facts as seen in (Zhang et al., 2023), aligns more closely with traditional practices. However, as illustrated in Table 1, these paradigms encounter significant challenges when applied to DocRE, particularly due to the complexity of handling samples that may contain multiple relations and a multitude of triplet facts.
In summary, current LLMs still exhibit significant gaps in performance for DocRE, indicating a need for further fine-tuning. Additionally, the existing RE templates, which treat relations as candidates, struggle to handle scenarios involving multiple relations. Coupled with the underwhelming effectiveness of current RE paradigms, there is a need for a paradigm shift.

heading: 3 Methodology

heading: 3.1 RE Paradigms
text: We summarized the existing paradigms of RE and designed a unique extraction paradigm, different RE paradigms are illustrated in Figure 2.
Document-facts (D-F). Fed with a document, the model directly outputs all triplets facts. This method is brute-force and requires the shortest inference time. It directly inputs relation types as candidates into the prompt and then let the model generate all triplet facts in one step as InstructIE (Wang et al., 2023b) did.
Document-relations-facts (D-RS-F). In this paradigm, the model extracts the relations present within the document and embeds all the predicted relations into the prompt to obtain triplet facts.
Document-relation-facts (D-R-F). In this framework, the model identifies relations within a given sentence and systematically traverses these relations to acquire triplet facts that correspond to each identified relation, which is similar to the approach taken by (Wei et al., 2023).
Document-relation-head-facts (D-R-H-F). In our newly designed paradigm, the model specifically focuses on each relation to identify an appropriate set of entities that will function as the 'head' in the triplet facts. Subsequently, the relevant triplets facts corresponding to these relations are extracted.
We test these paradigms with ChatGPT and the results are displayed in Table 1. We provide testing prompts for the Re-DocRED dataset under different paradigms using ChatGPT in Table 5. For brevity, we only provide two representative relation extraction prompt words. The rest are similar to these. We arrived at the following conclusion:
• LLMs still perform poorly in DocRE tasks involving the extraction of multiple relations
and triplet facts, achieving only single-digit scores. As of now, fine-tuning the model is still necessary.
• By extending the thought chain to derive final triplet facts, we can obtain more accurate triplet facts, though this approach does introduce a higher number of erroneous facts.
• Harnessing the last paradigm, which we name RHF, the model can find triplets facts more accurately in a step-by-step mode with finergrained tasks, thereby enhancing recall rates.

heading: 3.2 Dataset Processing
text: We used the Re-DocRED dataset for fine-tuning, refining it by removing duplicates and ensuring factual accuracy. This involved adjusting reciprocal relations like “follows” and “followed by” to accurately represent inversion, enhancing the dataset’s robustness and precision.
In earlier experiments with ChatGPT, we discovered that providing the model with descriptions
of relations enhances its capability to extract factual information. Nevertheless, incorporating Wikidata relation descriptions4 led to diminished performance, likely due to their occasional lack of clarity and precision, as exemplified by:
“located in the administrative territorial entity”: “The item is located on the territory of the following administrative entity. Use P276 for specifying locations that are non-administrative places and for items about events. Use P1382 if the item falls only partially into the administrative entity.”
Addressing this, we systematically rewrote all 96 relation descriptions, markedly improving model performance in Table 2. An example of our revised description is as follows. Details of all relation descriptions are presented in Table 6.
“located in the administrative territorial entity”: “This relation indicates that a subject (e.g., a place, event, or item) is situated within an administrative region, the object. Example: (Harvard University, located in the administrative territorial entity,
4https://www.wikidata.org/
Cambridge, Massachusetts).” Finally, in line with the RHF paradigm, we crafted instruction fine-tuning templates, breaking down the RE process of each sample into three distinct steps. The specific details of these templates can be found in Table 7, we provide a display of how we constructed our training data using simple prompt templates, the extraction of relations, the extraction of the head entity, and finally the triplet extraction.

heading: 3.3 QLoRA Tuning
text: Mistral-7B was selected as the foundation for fine-tuning because it demonstrated the best performance among the several open-source models tested when evaluating LLMs on the Re-DocRED task. To facilitate efficient training, we opted for PEFT’s QLoRA. The key advantage of QLoRA is its ability to combine the benefits of quantization and Low-Rank Adaptation (Hu et al., 2021), resulting in efficient fine-tuning. Specifically, quantization reduces data complexity, allowing for more efficient storage and processing, which is particularly valuable for deploying large models on resourceconstrained devices.
We leveraged three distinct QLoRA modules, each tailored to a specific stage of the RHF steps. This implementation was critical in enhancing RE efficiency. With the data volume varying across the intertwined tasks, a one-size-fits-all approach
could have compromised performance. However, the modular structure of QLoRA facilitated smooth integration with the underlying base model. As a result, we instituted three distinct QLoRA modules, each meticulously fine-tuned to its specific dataset. This meticulous approach resulted in the creation of the AutoRE, which amalgamates these modules for amplified DocRE performance.

heading: 4 Experiment

heading: 4.1 Experimental Setup
text: Test set. In our evaluation, we utilized the refined Re-DocRED test set consisting of 499 articles and 17,448 triplet facts, and a validation set encompassing 498 articles with 17,236 triplets, ensuring a comprehensive and precise assessment.
Evaluation Metric. We adopted the strict Micro F1 criterion, recognizing a prediction as correct only if it precisely captures the entire relation, along with both the head and tail entities. It’s important to highlight that within the Re-DocRED dataset, a triplet fact may encompass multiple aliases (mentions) for both the head and tail entities. Consequently, our evaluation protocol deems a prediction accurate if it correctly identifies any valid triplet pair. If a prediction aligns with any alias pair of the head and tail entity, it’s counted as correct, but alternate accurate aliases aren’t tallied in the correct statistics. Conversely, all incorrect predictions are flagged as false positives. This method ensures a stringent and statistically valid evaluation, lending robust credibility to the final results.

heading: 4.2 Overall Result
text: After training three distinct QLoRA modules, we test the performance on the Re-DocRED and then combine three QLoRAs to get the final performance on the dev set and test set, the result is shown in Table 3. When compared with TAG (Zhang et al., 2023) as a baseline which firstly reported the
end-to-end RE on Re-DocRED, our method has achieved SOTA results, as shown in Table 4. In both the dev set and test set, the performance improvement of AutoRE finetuned with Mistral-7B over TAG is approximately 7.44% on the dev set, and about 5.12% on the test set demonstrating the effectiveness of our approach. Furthermore, by decomposing the task into three subtasks and training with three LoRA modules, we not only achieved excellent results but also naturally acquired an easily extendable trait. This allows for targeted improvement of a specific module’s performance without impacting the performance of other subtasks. Additionally, it is worth noting that our work is the
first to utilize large language models for processing the Re-DocRED dataset. AutoRE can serve as a reference for subsequent research in this field.

heading: 4.3 Ablation
text: In the ablation study, we employed Mistral-7B to fine-tune the paradigms mentioned before, revealing that the RHF model yields the best performance when solely utilizing one QLoRA module. This finding substantiates our initial hypothesis during paradigm selection: employing a step-by-step approach enhances the extraction of triplet facts while significantly reducing erroneous triplets through fine-tuning. Building on this, we compared the im-
pact of including descriptions versus omitting them. The results confirmed that incorporating proper relation descriptions indeed benefits the model, as shown in Figure 4. Additionally, we explored the effectiveness of training the entire dataset with one QLoRA versus independently training different stages of RHF with three distinct QLoRAs. The latter approach demonstrated superior performance. We believe this is due to the data imbalance among predicting relations, predicting head entities, and predicting factual triples in the RHF paradigm, with the data volume for the three subtasks begin approximately 2.8%, 24.23%, and 72.97%, respectively. When combined, the model tends to favor the prediction of triples, while its capability to predict relations is relatively insufficient.
Additionally, we have applied this framework to Vicuna-7B and ChatGLM3-6B, and both models surpassed the current SOTA levels, demonstrating the universality of the AutoRE framework. The comparative results of these experiments are illustrated in the accompanying Figure 4. Vicuna-7B scored the highest, surpassing TAG by 10.03% and 9.03% respectively on the dev and test set, whereas ChatGLM3-6B was somewhat lower. This might be due to ChatGLM3-6B having a higher proportion of Chinese in its pre-training, while it was tested on an English task. We have deployed the system on the online platform5 for users to access and experience, as shown in Figure 3.

heading: 5 Conclusion
text: In this paper, we introduce RHF, a new paradigm for RE, alongside AutoRE, an advanced DocRE model. AutoRE represents a cutting-edge approach to the DocRE task, utilizing LLMs combined with QLoRA. This innovative model establishes a new standard, achieving SOTA results on the ReDocRED dataset. AutoRE proficiently addresses the intricate task of extracting multiple relations from document-level texts, a significant challenge that has stymied existing models. Our future goal is to create a comprehensive, unified framework for RE, fully leveraging the capabilities and promise of this paradigm.
Limitations
Insufficient Number of Relations. In real-world applications, the number of relations can reach
5https://models.aminer.cn/neptune/
thousands, significantly surpassing the 96 relations present in the Re-DocRED dataset. To better adapt to these real-world scenarios, it is imperative to gather more extensive datasets and expand the range of relations.
Limitation to In-Domain Data. AutoRE is not equipped to handle unseen relations. This limitation underscores the method’s inadequate generalizability, primarily due to the limited scope of data it has been trained on.

heading: Acknowledgements
text: This work is supported by Technology and Innovation Major Project of the Ministry of Science and Technology of China under Grant 2022ZD0118600, Natural Science Foundation of China(NSFC) 62276148 and 62425601, the New Cornerstone Science Foundation through the XPLORER PRIZE.

heading: Future Work
text: Our future goal is to create a comprehensive, unified framework for RE, fully leveraging the capabilities and promise of this paradigm. Limitations
Insufficient Number of Relations. In real-world applications, the number of relations can reach
5https://models. aminer. cn/neptune/
thousands, significantly surpassing the 96 relations present in the Re-DocRED dataset. To better adapt to these real-world scenarios, it is imperative to gather more extensive datasets and expand the range of relations. Limitation to In-Domain Data. AutoRE is not equipped to handle unseen relations. This limitation underscores the method’s inadequate generalizability, primarily due to the limited scope of data it has been trained on