On the Robustness of Document-Level Relation Extraction Models to Entity Name Variations (2024)

Abstract: Driven by the demand for cross-sentence and large-scale relation extraction, document-level relation extraction (DocRE) has attracted increasing research interest. Despite the continuous improvement in performance, we find that existing DocRE models which initially perform well may make more mistakes when merely changing the entity names in the document, hindering the generalization to novel entity names. To this end, we systematically investigate the robustness of DocRE models to entity name variations in this work. We first propose a principled pipeline to generate entity-renamed documents by replacing the original entity names with names from Wikidata. By applying the pipeline to DocRED and Re-DocRED datasets, we construct two novel benchmarks named Env-DocRED and Env-Re-DocRED for robustness evaluation. Experimental results show that both three representative DocRE models and two in-context learned large language models consistently lack sufficient robustness to entity name variations, particularly on cross-sentence relation instances and documents with more entities. Finally, we propose an entity variation robust training method which not only improves the robustness of DocRE models but also enhances their understanding and reasoning capabilities. We further verify that the basic idea of this method can be effectively transferred to in-context learning for DocRE as well.1

heading: 1 Introduction
text: The demand for cross-sentence and large-scale relation extraction has led to a surge of research interest in document-level relation extraction (DocRE), which aims to identify the relations between each pair of entities within a document based on the document context (Yao et al., 2019). While covering more realistic scenarios than its sentence-level
*Corresponding author. 1The data and code are available at https://github.com/
THU-BPM/Env-DocRE.
counterpart (Hu et al., 2023b), DocRE also brings new challenges, requiring a comprehensive modeling of interactions among different mentions of an entity, different entities and different entity pairs.
Recently, a series of DocRE studies propose various novel models and methods, continuously improving the performance on several DocRE benchmarks (Tan et al., 2022a; Zhou and Lee, 2022; Xiao et al., 2022; Sun et al., 2023). However, we observe that existing DocRE models may produce more erroneous predictions when we merely change the entity names in a test document. As illustrated in Figure 1, a well-trained DocRE model correctly extracts all four relation instances from the original document. However, after replacing the entity names in the document with a new set of names of the same entity types (e.g., change the song name Uptown Girl into another song name
16362
Endless Love), the model starts making mistakes, including both false positive and false negative predictions. This indicates that existing DocRE models may overly rely on entity information for extraction and lack robustness. Considering the vast and diverse space of entity names in real-world scenarios, which also expands constantly with numerous novel entity names, the poor robustness and generalization further impedes the reliable application of DocRE models.
As a result, we systematically study the robustness of DocRE models to entity name variations in this work. To audit the robustness of existing DocRE models, we first propose a general pipeline to automatically generate perturbed test documents with changed entity names. Building such a pipeline is non-trivial for three reasons: (1) The entity types are constrained by relation types in a fine-grained manner. For instance, the tail entity of relation record label in Figure 1 must be a record label. Therefore, the new entity name should not alter the original fine-grained entity type, otherwise the relation labels may no longer hold. (2) For an entity mentioned multiple times under different names, each alias should be replaced with a distinct name to exclude the interference caused by different coreference structures, like Sony Music ⇒ Matador Records and Sony⇒ Matador in Figure 1. (3) The introduced entity names should be of high quality and come from a wide range of sources. We strictly adhere to these three principles and design a four-stage pipeline based on Wikidata, which retrieves valid items from Wikidata for entity name substitution.
We further apply the proposed pipeline to DocRED (Yao et al., 2019) and Re-DocRED (Tan et al., 2022b), due to both being the largest and most widely used DocRE datasets, to create two novel benchmarks, named Env-DocRED and EnvRe-DocRED, for evaluating the robustness of DocRE models to entity name variations2. By conducting extensive experiments on both original and newly constructed benchmarks, we thoroughly evaluate the robustness of three representative DocRE models and two in-context learned large language models (LLMs). The results show that the performance of all evaluated models drops significantly on Env-DocRED and Env-Re-DocRED (e.g., the best model’s F1 drops from 79.3% on Re-
2Our proposed pipeline can also be applied or adapted to other DocRE datasets, which we discuss in detail in Section 9.
DocRED to 57.0% on Env-Re-DocRED), revealing the poor robustness to entity name variations. Further analyses reveal that the performance decline mainly lies in the increase of false negative predictions, and is more pronounced on cross-sentence relation instances and documents with more entities. We also analyze the reasons for performance drop by examining the loss of entity knowledge and name clues under entity name variations.
Finally, to improve the robustness of DocRE models to entity name variations, we propose an Entity Variation Robust Training method (EVRT) which is based on data augmentation and consistency regularization. For each training document, we generate a perturbed document by entity renaming. Then, in addition to the classification loss for entity pairs in the original document, our method introduces three extra objectives, which respectively penalize the classification errors for entity pairs in the perturbed document, the inconsistency between representations, and inconsistency between predictions of original and corresponding perturbed entity pairs. Experimental results demonstrate that EVRT not only improves the robustness of existing DocRE models but also enhances their understanding and reasoning capabilities. Besides, we transfer the idea of EVRT to in-context learning of LLMs and propose a simple prompt optimization strategy, which effectively enhances the robustness of in-context learning for DocRE.

heading: 2 Related Work
text: Document-Level Relation Extraction. Driven by the demand for cross-sentence and large-scale relation extraction, research on relation extraction has expanded from sentence level to document level (Quirk and Poon, 2017; Yao et al., 2019). Recently document-level relation extraction has attracted increasing research interest, with new models emerging constantly. Based on the way of modeling relational information from the context, existing studies can be categorized into graph-based and sequence-based approaches. The former typically abstract the document by graph structures and perform inference with graph neural networks (Zeng et al., 2020; Zhang et al., 2021; Wei and Li, 2022; Lu et al., 2023), while the latter encode the longdistance contextual dependencies with transformeronly architectures (Zhou et al., 2021; Xie et al., 2022; Zhang et al., 2022; Meng et al., 2023; Ma et al., 2023; Hu et al., 2024a).
Robustness and Entity-Related Robustness in NLP. Despite achieving great progress with large pre-trained language models in various tasks, modern NLP models are still brittle to out-of-domain data (Hendrycks et al., 2020), adversarial attacks (McCoy et al., 2019) or small perturbation to the input (Ebrahimi et al., 2018). Consequently, there has been a growing effort to explore robustness issues in NLP, such as building robustness evaluation benchmarks and proposing robustness enhancement strategies (Wang et al., 2022; Liu et al., 2024a,b; Hu et al., 2024b). One branch of works focus on entity-related robustness of NLP models. By introducing various types of perturbation to entity (names), previous works audit or improve model robustness on different tasks like named entity recognition (Lin et al., 2021), machine reading comprehension (Yan et al., 2022) and dialogue state tracking (Cho et al., 2022). Wang et al. (2023) analyse the behavior of relation extraction models under entity replacements. However, they focus on the task of sentence-level relation extraction and only consider person and organization entities.
Robustness of DocRE Models. Compared with other NLP areas, research on robustness in DocRE is relatively scarce. Xu et al. (2022) observe that DocRE models may err when non-evidence sentences of a document are removed and propose a sentence focusing loss to improve the robustness. Chen et al. (2023) reveal the poor robustness of DocRE models to word-level attacks such as synonym substitution. A few recent works also construct entity-level attacks to investigate the robustness of DocRE models (Li et al., 2023; Chen et al.,
2023). However, all these attacks are not natural or adversarial, as they either disrupt entity structures (e.g., random entity mention drop) or alter entity types (e.g., random out-of-distribution entity substitution from a very limited source), rendering partial relation labels no longer valid. In contrast, we propose a principled pipeline to generate entityrenamed documents with labels preserved, and systematically evaluate and improve the robustness of DocRE models to entity name variations.

heading: 3 Problem Formulation
text: Given a document D which contains a set of entities E = {ei}Nei=1, the task of document-level relation extraction is to predict the set of all possible relations between each entity pair (eh, et) ∈ {(ei, ej) | i, j = 1, . . . , Ne; i ̸= j} from a predefined relation type set R. The subscripts of eh and et refer to the head and tail entity in an entity pair. An entity ei can occur multiple times in the document via Nei mentions Mei = {mij} Nei j=1, where the mention mij refers to the token span of ei’s j-th occurrence in the document.

heading: 4 Benchmark Construction
text: In this section, we elaborate on the process of constructing benchmarks for evaluating the robustness of DocRE models to entity name variations. We first propose a general pipeline to generate documents with changed entity names, then apply the pipeline to DocRED and Re-DocRED to create the Env-DocRED and Env-Re-DocRED benchmarks.

heading: 4.1 Construction Pipeline
text: As shown in Figure 2, our proposed pipeline consists of the following four steps.
Step 1: Entity Linking Based on Wikidata. Given a document, we first link each entity in the document to an item in Wikidata. Each item in Wikidata has a label and any number of aliases, and is uniquely identified by a number starting with “Q”. For example, we link the entity Westlife to item Westlife(Q151241) in Wikidata. Depending on the dataset at hand, we can perform entity linking using Wikidata Search API, off-the-shelf tools or methods specifically optimized for the datasets.
Step 2: Fine-grained Entity Typing. Next we query the value of Instance Of property (numbered as P31 in Wikidata) for each linked item on Wikidata, to obtain the fine-grained type of each entity, like boy band(Q216337) for Westlife(Q151241) in Figure 2.
Step 3: Alias-count-matched Candidate Entity Retrieval. Based on the fine-grained type of each entity, we further retrieve additional Wikidata items with the same entity type as candidates by executing a reverse query of Step 2. Note that we only retain those items whose number of aliases (plus label) is greater than or equal to the number of aliases of the original entity in the document. For example, since the entity Sony Music is mentioned under two different names in the document, we only take the retrieved items of record label with at least one Wikidata alias.
Step 4: Alias-wise Entity Mention Name Substitution. Finally, for each entity in the document, we randomly select an item from its candidate set and use this item to perform alias-wise entity mention name substitution, i.e., substitute a distinct name of the item for each alias of the original entity, like Sony Music⇒ Matador Records and Sony⇒ Matador in Figure 2.

heading: 4.2 Env-DocRED and Env-Re-DocRED Benchmarks
text: With the proposed pipeline, we further construct the robustness evaluation benchmarks based on existing datasets, which we choose DocRED (Yao et al., 2019) and Re-DocRED (Tan et al., 2022b) in this work. DocRED is one of the largest and most popular public datasets for DocRE, which is collected from English Wikipedia doc-
uments. DocRED has 96 pre-defined relation types, along with 3053/1000/1000 documents for training/development/test. Each document in DocRED has 19.5 entities and 12.5 relation triples on average. Re-DocRED is a revised version of DocRED, resolving the missing relation issue in DocRED. The 3053 revised training documents contain 28.1 triples on average and 1000 revised development documents (split into 500/500 development/test documents) have 34.7 triples on average.
We iterate over the development and test sets of DocRED and Re-DocRED and apply the pipeline five times on each document with different random seeds. We name the two newly constructed benchmarks Env-DocRED and Env-Re-DocRED, with the former having 3053/5000/5000 and the latter having 3053/2500/2500 documents for training/development/test. We adopt the entity linking method and results of Genest et al. (2023) in Step 1, which has a high quality benefited from its specific design for DocRED and Re-DocRED. Besides, since all entities of NUM and TIME type in (Re)DocRED can not be linked to Wikidata, we take a rule-based substitution method to produce novel names for number and time. Although a small portion of entities remain unlinked, statistics (Table 1) show that we have altered the name of over 92% entities in original datasets.

heading: 5 Robustness Evaluation and Analysis
text: In this section, utilizing the constructed benchmarks, we conduct a comprehensive evaluation and analysis on the robustness of existing DocRE models to entity name variations.

heading: 5.1 Selected Models and Evaluation Metrics
text: We choose three public-available DocRE models which are representative for their strong performance and high popularity. DocuNet (Zhang et al.,
2021) formulates DocRE as a semantic segmentation task and captures both local context information and global interdependency among triples for extraction. KDDocRE (Tan et al., 2022a) uses an axial attention module for two-hop relation reasoning and an adaptive focal loss to address the class imbalance problem. NCRL (Zhou and Lee, 2022) shares the same model with a strong DocRE baseline ATLOP (Zhou et al., 2021) but improves upon the learning of none class. We use Ign F1 and F1 scores as the evaluation metrics, where Ign F1 measures the F1 excluding those relational facts shared by the training and development/test sets. For each model, we all experiment with BERTbase (Devlin et al., 2019) and RoBERTalarge (Liu et al., 2019) encoder, leading to six submodels. We reimplement all models with their official codes and report the the mean and standard deviation results by five trials with different random seeds. Since the test set of DocRED is released by Codalab, we report the official test score of the best checkpoint on development set.

heading: 5.2 Main Evaluation Results
text: We present the evaluation results on the test sets of four benchmarks in Figure 3. We can observe that all DocRE models have a significant performance fluctuation on Env-DocRED and Env-Re-DocRED, with the relative F1 drop ranging from 21% ~31%, revealing the insufficient robustness to entity name variations. Model-wise, we find that the three selected DocRE models show similar relative decline in performance, with none being significantly more robust than others. Encoder-wise, we find that RoBERTalarge with higher performance also
leads to better robustness than BERTbase. Datasetwise, somewhat surprisingly, the relative decrease in F1 is even larger on Env-Re-DocRED than EnvDocRED. This suggests that despite Re-DocRED providing more complete relation labels, DocRE models still fail to gain benefits in robustness.

heading: 5.3 Further Analysis
text: In order to gain more in-depth insights, we conduct further analysis by answering the following questions.

heading: Q1: What is the performance bottleneck of DocRE models under entity name variations?
text: Given that the entity name variations lead to a drop in performance, a natural question is whether the model generates more false positive or false negative predictions. To better understand the performance bottleneck of DocRE models, we compare the changes in precision and recall of three models with BERTbase encoder. As shown in Table 2, the recall across models decreases significantly, while the precision changes little and even increases on Env-DocRED. This indicates that false negative predictions dominate the poor robustness to entity name variations.

heading: Q2: Do models show poorer robustness when predicting inter-sentence relations?
text: Since a major feature of DocRE is to extract the complex cross-sentence relations, we further analyse models’ robustness in predicting intra-sentence and inter-sentence relations. We report the Intra F1 and Inter F1 of three BERTbase encoded DocRE models in Table 3, which respectively evaluate on the entity pairs with and without mentions in same sentence. We can observe that on both Env-DocRED and Env-Re-DocRED, the relative F1 drop for inter-sentence relations is approximately twice that of intra-sentence relations, which indicates that existing DocRE models show poorer robustness to entity name variations when predicting inter-sentence relations.

heading: Q3: How does the model robustness vary with the number of entities in the document?
text: We also investigate the robustness of DocRE models on documents with varying number of entities. This aids in better extrapolating our findings to longer documents, which often contain more entities. We divide the documents into different groups by the number of entities and evaluate the performance on each group. We showcase the results of NCRL-BERTbase in Figure 4. As the number of entities increases, the absolute performance drop under entity name variations gets larger, especially on Env-Re-DocRED. The slopes of the linear fits on DocRED, Env-DocRED, Re-DocRED, Env-ReDocRED are -0.35, -0.42, -0.24 and -0.69 respectively. Note that the performance itself also shows a decreasing trend when encountering more entities, thus the relative performance drop should be
more significant. This suggests that the model may be more brittle as the number of entities increases.
Q4: How can we disentangle the reasons for the performance drop?
Yan et al. (2022) pointed out that the information associated with the entity name that can be leveraged by the model includes both entity knowledge and name clues. The former refers to the world knowledge associated with the entity like “Westlife is a famous boy band”, which mainly comes from pre-training. The latter refer to the statistical clues associated with the name’s surface form like “Westlife always appears with the performer relation in training set”, which mainly come from fine-tuning. The perturbations to entity names may break these two types of information.
We adopt two measurements to better understand the information loss. We calculate the popularity of entities (Huang et al., 2022), i.e., how many times the linked item of the entity appears in a relation instance in Wikidata, in each benchmark’s test set to roughly quantify the entity knowledge. As shown in Table 4, the popularity of entities in two new benchmarks drops significantly. For name clues, we calculate the percentage of entity mentions that appear in training sets for each benchmark’s test set. As shown in Table 5, the proportion also has a noticeable drop in two novel benchmarks.

heading: Q5: How robust is in-context learning of LLMs under entity name variations?
text: Recently large language models (LLMs) (Brown et al., 2020) have achieved promising few-shot results on many tasks through in-context learning (ICL) (Dong et al., 2023). Therefore, we also conduct an experiment to explore how robust of ICL for DocRE under entity name variations. We use gpt-3.5-turbo-01253 and gpt-4-0125-preview4 due to them being the most capable LLMs currently. We experiment on both 1-Shot and 3-Shot settings, which represent providing 1 and 3 example document(s) and gold relation instances as demonstrations. We randomly select demonstration document in the training set for each test document and set the temperature parameter to 0 for least randomness. The detailed prompt template is presented in Appendix A. The experimental results on test sets of Re-DocRED and Env-Re-DocRED are shown in Table 6. We find that on both settings, the two incontext learned LLMs show a performance drop on Env-Re-DocRED, suggesting that the robustness issue exists not only in specialized models but also in large models.

heading: 6 Entity Variation Robust Training
text: Due to the unsatisfactory robustness of existing DocRE models to entity name variations, we further explore the method for enhanced robustness. Intuitively, we can adopt a similar approach as the proposed pipeline to perturb each training document with a group of new entity names. The derived document naturally shares the same relation labels with the original one. Also, a robust DocRE model should generate consistent representations and predictions for each corresponding entity pair in the original and perturbed documents. Based on such motivation, we propose an entity variation robust training method (EVRT) that is enhanced by
3https://platform.openai.com/docs/models/gpt-3-5-turbo 4https://platform.openai.com/docs/models/gpt-4-and-
gpt-4-turbo (Due to limited budget, the experiments with gpt-4-0125-preview only use 1/5 documents.)
data augmentation and consistency regularization. Specifically, given a labeled entity pair (eh, et) in a document, vanilla approaches typically train the DocRE model with a classification objective Lclo = ℓtask(eh, et), where ℓtask denotes the loss function depending on the specific model.
Denoting the corresponding entity pair of (eh, et) in the perturbed document as (eĥ, et̂), our proposed method first incorporate the classification loss Lclp = ℓtask(eĥ, et̂) for (eĥ, et̂) to penalize the classification errors for entity pairs in the perturbed document. Then we introduce representation consistency regularization and prediction consistency regularization to encourage the model to produce consistent representations and predicted probability distributions between (eh, et) and (eĥ, et̂). Formally, we define the representation consistency regularization loss as:
Lrcr = ∥z(h,t) − z(ĥ,t̂)∥22, (1)
where z(h,t) is the pair representation of (eh, et). And we define the prediction consistency regularization loss as:
Lpcr = ∑
r∈R DSKL(p(h,t)r ,p(ĥ,t̂)r ), (2)
where p(h,t)r = [P (h,t) r , 1−P (h,t)r ], P (h,t)r is the predicted probability of relation r for (eh, et), DSKL is the symmetric KL divergence:
DSKL(p, q) = DKL(p∥q) +DKL(q∥p), (3)
where DKL is the vanilla KL divergence. The overall objective is defined as:
L = Lclo + Lclp + αLrcr + βLpcr, (4)
where α and β are two hyperparameters. Note that to prevent the incorporated novel entity names for training document perturbation have overlap with those entity names for substitution when constructing the benchmarks, resulting in potential shortcuts, we isolate the new entity names introduced during benchmark construction when replacing the entities in training documents.

heading: 7 Experiments

heading: 7.1 Main Results
text: The main results on the test sets of four benchmarks are shown in Table 7. It is shown that when equipped with the proposed EVRT method, all
DocRE models achieve a significant performance gain on Env-DocRED (a maximum more then 9% absolute increase in F1) and Env-Re-DocRED (a maximum more than 12% absolute increase in F1). Meanwhile, the performance on DocRED and ReDocRED only shows a slight drop. All these results indicate that EVRT can effectively improve the robustness of existing DocRE models to entity name variations.

heading: 7.2 Ablation Study
text: We further conduct an ablation study on EnvDocRED and Env-Re-DocRED to investigate the influence of three newly added training objectives. As shown in Table 8, only introducing one of Lclp, Lrcr and Lpcr has lead to a significant performance improvement, which indicates the effectiveness of each objective. When combining these losses pairwise, the performance is further enhanced. And the
best performance is achieved when simultaneously using three objectives together. We also observe that compare to Lrcr, Lclp and Lpcr may play a more important role for the improvement.

heading: 7.3 Understanding and Reasoning Capability Evaluation
text: We also take the MAP evaluation metric proposed in Chen et al. (2023) to evaluate the understanding and reasoning capabilities of DocRE models trained with and without our EVRT method. Given top K words with the highest attribution values, the formula of MAP over T relational facts is:
MAP(K) = 1
T
T∑
t=1
APt(K) = 1
T
T∑
t=1
1 K
K∑
i=1
Pt(i) ·1t(i),
(5)
where 1t(i) is the indicator function of the i-th important word for predicting the t-th relational fact. We select all possible values of K and report the MAP curve of NCRL-BERTbase and
NCRL-BERTbase + EVRT models in Figure 5. It is observed that the MAP values of NCRLBERTbase + EVRT are consistently higher than NCRL-BERTbase, suggesting that the proposed EVRT method not only improves the robustness of DocRE models but also enhances their understanding and reasoning capabilities.

heading: 7.4 Entity Variation Robust In-Context Learning
text: The results in Section 5.3 indicate that utilizing in-context learning of LLMs for DocRE also shows insufficient robustness to entity name variations. A natural question is can we transfer the basic idea of EVRT to improve the robustness of in-context learning. We conduct a preliminary attempt by designing a simple entity variation robust in-context learning method, which optimizes the prompt with demonstration augmentation (DA) and consistency guidance (CG). Based on the vanilla prompt, demonstration augmentation adds an entity-renamed document for each original demonstration document. And consistency guidance further expands the prompt by explicitly explaining that “The only difference between two documents lies in the entity names. Apart from the entities, the contextual content of the two documents is entirely the same. Therefore, the expected outputs for the two documents are also identical. When extracting relation triples from the test document, please base the extraction on the context of the document and avoid identifying the relations solely based on the information of the entities themselves.”. As shown in Table 9, this simple strategy also effectively enhances the robustness of LLMbased in-context learning methods.

heading: 8 Conclusion
text: In this work, we systematically study the robustness of DocRE models to entity name vari-
ations. Our main contributions are three-fold: (1) Resource-wise, we propose a general pipeline to reasonably generate entity-renamed documents and construct two novel benchmarks, Env-DocRED and Env-Re-DocRED, for robustness evaluation. (2) Experiment-wise, we conduct comprehensive experiments on multiple DocRE models to evaluate their robustness and provide further analyses from multiple perspectives. (3) Methodology-wise, we propose entity variation robust training and incontext learning methods, effectively improving the robustness of DocRE models. We hope our work can benefit and offer insights for future research to develop more robust DocRE models.

heading: 9 Limitations and Future Directions
text: In this section, we analyse the limitations of our work from three perspectives and hope to provide inspiration for future works.
Task Setting. Our study is grounded upon a classic setting of DocRE where the entity information including entity mention positions and coreference clusters of mentions are given beforehand. Some recent works explore the end-to-end setting of DocRE, which requires the model to jointly perform mention detection (and optionally classification), coreference resolution and relation extraction, aligning better with real-world application scenarios (Eberts and Ulges, 2021; Giorgi et al., 2022; Xu and Choi, 2022; Zhang et al., 2023). Investigating the robustness of end-to-end DocRE approaches to entity name variations is a promising direction for future works. More importantly, since the proposed pipeline for entity name substitution does not alter entity types and coreference labels, our constructed benchmarks can be directly utilized for the study of end-to-end DocRE model robustness, rendering the two benchmarks more valuable.
Dataset Domain and Language. Given that we construct the robustness evaluation benchmarks based on DocRED and Re-DocRED, which originate from English Wikipedia documents, our findings may be somewhat limited to English, generic-domain scenarios. Leveraging other wellestablished DocRE datasets, future works are encouraged to extend the study on entity name variation robustness of DocRE models to more domains such as news (Zaporojets et al., 2021), biomedicine (Li et al., 2016), social media (Hu et al., 2023a) and scientific publications (Luan et al., 2018), and more
languages such as Chinese (Cheng et al., 2021) and Korean (Yang et al., 2023). As Wikidata covers a wide range of domains and languages, the proposed benchmark construction pipeline can also be applied to other datasets. For datasets that are hard to be linked to Wikidata, one may explore the possibility of adapting the pipeline with an appropriate knowledge base.
Methodology. Since the proposed entity variation robust training and in-context learning frameworks generate a perturbed document with changed entity names for each training document, finetuning pre-trained models incurs larger memory overhead, and utilizing large language models for in-context learning entails higher time and cost expenses. Additionally, although the proposed methods significantly improve the performance of multiple models on Env-DocRED and Env-Re-DocRED, there is still a certain gap compared to DocRED and Re-DocRED. An intriguing avenue for future research is to explore more efficient and effective techniques to improve the robustness of DocRE models to entity name variations.

heading: Future Work
text: In this section, we analyse the limitations of our work from three perspectives and hope to provide inspiration for future works. Task Setting. Our study is grounded upon a classic setting of DocRE where the entity information including entity mention positions and coreference clusters of mentions are given beforehand. Some recent works explore the end-to-end setting of DocRE, which requires the model to jointly perform mention detection (and optionally classification), coreference resolution and relation extraction, aligning better with real-world application scenarios (Eberts and Ulges, 2021; Giorgi et al. , 2022; Xu and Choi, 2022; Zhang et al. , 2023). Investigating the robustness of end-to-end DocRE approaches to entity name variations is a promising direction for future works. More importantly, since the proposed pipeline for entity name substitution does not alter entity types and coreference labels, our constructed benchmarks can be directly utilized for the study of end-to-end DocRE model robustness, rendering the two benchmarks more valuable. Dataset Domain and Language. Given that we construct the robustness evaluation benchmarks based on DocRED and Re-DocRED, which originate from English Wikipedia documents, our findings may be somewhat limited to English, generic-domain scenarios. Leveraging other wellestablished DocRE datasets, future works are encouraged to extend the study on entity name variation robustness of DocRE models to more domains such as news (Zaporojets et al. , 2021), biomedicine (Li et al. , 2016), social media (Hu et al. , 2023a) and scientific publications (Luan et al. , 2018), and more
languages such as Chinese (Cheng et al. , 2021) and Korean (Yang et al. , 2023). As Wikidata covers a wide range of domains and languages, the proposed benchmark construction pipeline can also be applied to other datasets. For datasets that are hard to be linked to Wikidata, one may explore the possibility of adapting the pipeline with an appropriate knowledge base. Methodology. Since the proposed entity variation robust training and in-context learning frameworks generate a perturbed document with changed entity names for each training document, finetuning pre-trained models incurs larger memory overhead, and utilizing large language models for in-context learning entails higher time and cost expenses. Additionally, although the proposed methods significantly improve the performance of multiple models on Env-DocRED and Env-Re-DocRED, there is still a certain gap compared to DocRED and Re-DocRED. An intriguing avenue for future research is to explore more efficient and effective techniques to improve the robustness of DocRE models to entity name variations