Anaphor Assisted Document-Level Relation Extraction
Authors: Chonggang Lu
Richong Zhang
Kai Sun
Jaein Kim
Cunwang Zhang
Yongyi Mao
Venue: EMNLP 2023 Main (2023)
Keywords: relation extraction
document level
anaphor
graph

Abstract: Document-level relation extraction (DocRE) involves identifying relations between entities distributed in multiple sentences within a document.
Existing methods focus on building a heterogeneous document graph to model the internal structure of an entity and the external interaction between entities. 
However, there are two drawbacks in existing methods. On one hand, anaphor plays an important role in reasoning to identify relations between entities but is ignored by these methods. On the other hand, these methods achieve cross-sentence entity interactions implicitly by utilizing a document or sentences as intermediate nodes. Such an approach has difficulties in learning fine-grained interactions between entities across different sentences, resulting in sub-optimal performance. To address these issues, we propose an Anaphor-Assisted (AA) framework for DocRE tasks. Experimental results on the widely-used datasets demonstrate that our model achieves a new state-of-the-art performance.

Anaphor Assisted Document-Level Relation Extraction
Chonggang Lu', Richong Zhang' 1.2+ Kai J Kim', Cunwang Zhang', Yongyi Mao SKLSDE, Beihang University, Beijing. China *Zhongguancun Laboratory, Beijing, China
Sccooo of Electrical Engineering and Computer Science, University of Ottawa, Ottawa, Canada
(lucg, zhangre] wwt..baa..ed..co (sunkai, jaeinn jaeiaiaaaiinnnnnnea.oaa zhangow) buaa.edu.en, ymao@uottawa.co
Abstract
[1] "Nie "is the ninth episode of the third season of the American
television series X Filex ...[3] [3] was by
Document-lever relation extraction (DocRE) involves identifying relations between entities distributed in multiple sentences within a doc- ument. Existing methods focus building heterogeneous document graph to model the internal structure an entity and the external interaction between entities. However, there are two drawbacks in existing methods. On one hand, anaphor plays an important role in reasoning to identify relations between en- tities but is ignored by these methods. On the other hand, these methods achieve cross- sentence entity interactions implicitly by uti- lizing a document or sentences as intermedi- ate nodes. Such an approach has difficulties in learning fine-grained interactions between entities across different sentences, resulting in sub-optimal performance. To address these is- sues, we propose an Anaphor-Assisee (AA) framework for DocRE tasks. Experimental re- sults on the widely-used datasets demonstrate that our model achieves a new state-of-the-at performance.
David Notter,
by Cheis Cartez, Frank Spotcitz Howard
Gordon... ...18] The show centers P'Bssppcial agents For Mukler
David Dochovny ) and Dana. Scully Gillian Anderson ) who work...
The X Filex
Die show
Fox Mulder
creator
Niver
Subject: Fox Mulder
Relation: creater
Object: Chris Carter
Figure 1: An example from DocRED dataset.
et al., 2019) and RoBERTa (Liu et al., 2019), to cap- ture the long-range dependency between entities in the document, which are called the transformer- based models (Zhou et al., 2021; Zhang et al., 2021; Xiao et al., 2022a; Tan et al., 2022a). These mod- cls only take the word sequence input without considering the internal structure of an entity and fail to explicitly learn the external interaction be- tween entities, thus having difficulties in address- ing complex instances that require reasoning (Zhou et al., 2021; Xie et al., 2022). To enhance the rea- soning ability of transformer-bate models, recent works propose to transfer the document into graph where entity mentions are introduced nodes and operate graph neural network (Scarselli al., 2008) on the document graph to explicitly learn information interactions between entities, which are usually called the graph-based models (Wang et al., 2020c; Zeng et al., 2020; Wang et al., 2020a).
1 Introduction
Document-level relation extraction (DocRE) has garnered increasing attention from researchers lately due to its alignment with real-world appli- cations, where large number of relational facts are expressed multiple sentences (Yao et al., 2019). Compared with its sentence-level counter- part (Zhang et al., 2018; Zhu et al., 2019; Sun et al., 2020), DocRE is more difficult as it requires more sophisticated understanding of context and needs model the interaction in mentions belonging to the same different entities distributed in multiple sentences (Yu al., 2022; Xu et al., 2022, 2021).
Though some promising results have been achieved by recent graph-based models, cross- sentence entity interaction is not well learned in these models since most of them use document node (Zeng et al., 2020) or sentence nodes (Wang et al.. 2020a) as intermediate nodes link entities in different sentences with an unclear information delivery. Such an approach captures interactions between entities distributed different sentences
To address DoCRE, most of existing works use pre-trained language model, such as BERT (Devlin
"Corresponding author.
!Our cnde available at https: sttps://githut com/ BurgerrurrerBurgge//A.

---

via a common document node or sentence node, which is not effective learning fine-gratd ene-grained in- teractions between entities, leading to sub-optimal performance. Besides, previous methods (Zhou et al., 2021; Zeng et al., 2020; Ma et al., 2023) are generally missing the anaphors in document, However, identifying cross-sentence relations be- tween entities often requires the intervention of anaphors since anaphors often convey the inter- action between entities in multiple sentences as intermediate nodes, such as pronouns or definite referents, which refer to entities within the context.
out irrelevant information. Extensive experiments on DocRED and Re-DocRED confirm the effec- tiveness of our proposed anaphor-assisted model. In summary, the main contributions of this paper are as follows:
We propose novel framework that explicitly and jointly models coreference and anaphora information, enabling the capture of fine- grained interactions between entities.
We employ a dynamic algorithm for graph pruning and structure optimization, which ne- cessitates minimal additional annotations.
As shown in Figure 1, to identify the relation "creator" between head entity "Fox Mulder" and tail entity "Chirs Carter", sentences [1], [3], and [8] should be considered simultaneously. In these sentences, the anaphors play an important role in helping the model identify the target relation. Specifically, "it" in sentence [3] refers "Nisei" in sentence [I] while "The show" in sentence [8] refers "The X-Files" in sentence [1]. With these anaphors as bridges, the information transport be- tween entities across sentences can be facilitated. For instance, the connection between "Fox Mulder" and "The X-Files" can be established via "The show". Similarly the connection between "Nisei" and "Chirs Carter" can be established via "ir". By leveraging the correspondence between these anaphors and entities, is casier for the model to capture the semantic relations between "Fox Mulder" and "The X-Files". "Nisei" and "Chirs Carter", and finally promote extraction of the rela- tion between "Fox Mulder" and "Chirs Carter".
Experimental results show that our approach is valid and outperforms previous statc-of- the-art document-level RE methods on two DocRE datasets.
2 Related Work
According to whether explicitly model the infor- mation interaction between entities, current works can be divided into two lines: transformer-based models and graph-based models.
2.1 Transformer-based Models
Transformer-based models take only the word quence of a document input and leverage the transformer (Vaswani al., 2017) to implicitly capture the long-range contextual dependencies be- tween entities. So far, lots of transformer- based models have been proposed for DocRE. major- ity of these models focus on extracting more ex- pressive entity representations from the output of transformer (Tan et al., 2022a; Xiao et al., 2022b; Zhou al., 2021). Among these works, Zhou et al. (2021) propose localized context pooling to hance the representations of entities by locating the relevant context. Xie et al. (2022) propose an evidence-enhanced framework. EIDER, that effec- tively extracts and fuses evidence in the inference stage while Ma et al. (2023) uses evidence to con- struct supervision signals for model training with the aim to filter out irrelevant information. How- ever, without considering the internal structure of an entity or explicitly learning the external inter- action between entities, these transformer-bast models are found have difficulties identifying relations some complex instances that require reasoning (Zhou et al., 2021; Xie et al., 2022).
Based on these observations above, we are mo- tivated to develop a new framework to explicitly and jointly leverage the coreference and anaphora information in the document. We achieve this by introducing new document graph which is con- structed by considering all possible entity mentions and anaphors in the document. To distinguish the importance of nodes and edges in the graph. we define three types of edges to connect the nodes and further propose an attention- graph con- volutional neural network to dynamically learn the structure of the graph. With the proposed frame- work, the information transport the graph is sufficiently modeled and an expressive entity pre- sentation is extracted for final classification. Fol- lowing previous works (Ma et al., 2023; Xiao et al., 2022a; Xie et al., 2022), we also introduce evidence retrieval as an auxiliary task to help the model filter

---

advmou
attr
paza
pob)
del
09e
cnddy
prop
det
There
is
Walmari
next
Tom
Hit
warks
al
the
market.
PRON
AUX
DET
PROPN
ADV
ADP
PROPN
PART
NOUN
PRON
VERB
ADP
DET
NOUN
Figure 2: Dependency Tree and Part-of-Speech Tags for an example.
2.2
Graph-based Models
3 Problem Definition
Graph-based models focus on building a document graph and explicitly learn the information inter- action between entities based on the constructed graph. Most existing works define three types of nodes: mentions, entities, and sentences, and link nodes using heuristic rules, such as connecting mentions within the same entity or mentions within the same sentence but belonging to different enti- ties (Li et al., 2020; Zhao et al., 2022; Sahu et al., 2019; Wang et al., 2020a; Zeng et al., 2020). For example, Christopoulou et al. (2019) introduces three types of nodes (mentions, entities, and sen- tences) to build the document graph and propose an edge-oriented graph neural network operate over the constructed graph. Wang al. (2020b) build a document graph similar to (Christopoulou et al., 2019) but design a global-to-locee mechanism to encode coarse-grained and fine-grained semantic information of entities. Li et al. (2020) build het- erogeneous document graph that is comprised of entity nodes and sentence nodes with three types of edges: centence-sentente edges, entity-entity edges, and entity-sentence edges. Zeng al. (2020) propose to aggregate contextual information using a mention-level graph and develop a path reasoning mechanism to infer relations between entities.
Consider document D containing tokens TD (4)_11 sentences Sp - { and entities ED [c]] the goal of document-level rela- tion extraction is to predict the relation r for each entity pair (en, et) from a pre-defined relation set R U {N A}. where {NA} denotes no relation be- tween two entities. Each entity
Ep is repre- sented by its mentions M, = [ and each mention m MM is a phrase in the document. Except for the mentions, there are other phrases, known as anaphors, that may refer to the entities. Anaphors usually include pronouns like he, she, or it, and definite referents like the song or the show. Our purpose is use the information of anaphors to promote relation extraction. Morcover, for each entity pair that has a valid relation r R, a of evidence sentences Vh, E Sp is provided to spec- ify the key sentences in the document for relation extraction.
Model
4.1 Anaphor Extraction
We use the off-the-shelf NLP tool, Spacy² iden- tify anaphors. Specifically, we employ Spacy's 's tag- ger functionality for Pfrt-of-Speech tagging, treat- ing all pronouns (PRON) as potential anaphors. Additionally, we utilize Spacy's 's dependency parser to aid in anaphor identification. In particular, when token's 's dependency relation is 'det' (indicating determiner) and its text is 'the,' our approach in- volves identifying all tokens positioned between this specific 'the" token and its associated head as an anaphor. Figure 4.1 shows an example in the sentences 'There a Walmart next to Tom's house. He works at the market.' In this example, 'He' and 'the market' are extracted as anaphors.
However, these graph-based models focus on learning interactions between entities in the same sentence while interactions between entities in dif- ferent sentences are not well modeled, thus hav- ing difficulties in handling the case shown in Fig- ure Besides, these models generally construct a static graph, assuming different edges are equally important. Although Nan et al. (2020) propose to treat the document graph as a latent variable and induce it based on attention mechanisms, their method ignores the co-referential information of entities and mentions, leading sub-optimal per- formance. More importantly, previous methods generally overlook anaphors in the document, like he, she, it, and definite referents, which play an im- portant role cross- sensene relation extraction.
hhtts:://ppcy.io/

---

[1] Nisel "is the ninth episode of the third scason of the American science fiction television series The Fille..
[3] It was directed by David Nutter, and written by Chris Carter, Frunk Spotnitz and Howard Gerdon.
[8] The show centers on EBI special agents Fox Mulder David Duchovny
Scully Gillian Anderson who
Construct
Training Stage
Inference Stage
CheeClana
Model Ano
Wirsetf
GCN
N head
Maller
Dynamie
Pfrle DO
Uie
lterate
Toe dlowe
The File
[]] Nisi the....
Encoder
Select Nodes
COO
GCN
N head
Bilinear
Tbcabes..
Fusion
Representations
frre
D'
Construct
Hy
Ph
Paur
Sum Sentence
PaLs
P(r opi
[1] Nissiill spoodk
l by Duvld
Ps
Modelw Fevt
Evidence Score
Lev
ESt Marde
Figure 3: The overview of our anaphor DocRE framework.
4.2 Document Graph Construction
nouns and definite referents that represent an entity. To exploit them, we create anaphor nodes for each pronoun and definite referent in the document. For example, in Figure 1, the pronoun it and the definite referent The show refer to Nisei and The X-Files and are represented anaphor nodes in the graph.
We transfer the document into an undirected graph G = (V, F)) which is constructed by considering all mentions and possible anaphors in the document. Specifically, define two types of nodes: mention nodes and anaphor nodes. To distinguish the se- mantics of connections between different types of nodes, we introduce three types of edges; mention- anaphor edge, co-reference edge, and inter-entity edge. These edges represent anaphora resolution, coreference resolution, and external entity interac- tion respectively.
4.2.2 Graph Edges
To this end, obtained node set V composed of the mention and anaphora nodes. We then make connections between these nodes obtain the edge F. Specifically, there are three types of edges: mention- anaphor mention--nnaphrr edges, co-reference edges, and inter enter- edges.
4.2.1 Graph Nodes
We collect two types of phrases in the document build the node set V, mentions and anaphors. Mentions are annotated in the document while the anaphors are obtained by a semantic parsing tool. Mention Nodes: Entities are represented by men- tions in the document. In the graph, each mention is represented by mention node. For example, in Figure 1, the mentions { 'Niseii,, Chris Carter, Fox Mulder) are represented as mention nodes in the graph. Note that all mentions are annotated in datasets, and one entity may have multiple men- tions.
Edges: First, anaphor nodes should be associated with their corresponding men- tion nodes and there should be mention-une mention-anaphor edges between them. However, the ground truth correspondence between mention and anaphor nodes unknown. Thus, we consider all possible connections between the mention and the anaphor nodes. More specifically, we link each mention node with all anaphor nodes.
Mention-Anapee
Co-reference Edge: Mentions that refer to the same entity connected with each other. This allows modeling the internal structure of an en- tity as well as facilitating the information transport between multiple mention nodes of the entity to extract an expressive entity representation.
Anaphor Nodes: Besides the mentions, there are other linguistic components that may refer to an entity. Specifically, in a document, there are pro-

---

To enhance information inter- action between different entities in the document, we connect every pair of mention nodes that refer to different entities across the entire document. In other words, if two mentions refer to different en- tities, an edge exists between the corresponding mention nodes in the graph.
Inter-Entity Edges:
of [0.1]. Aij represents the importance of the edge between two connected nodes v and P), where a high Ai suggests a strong relationship between the two nodes otherwise weak. Formally, Á is computed as follows:
HA x (HyW
Á softmar Au
vd
4.3 Document Encoding
(4)
Given a document D containing tokens TD {t+)/=1- we insert a special token "*" before and behind each mention as an entity marker (Zhang et al., 2017), with each mention m, represented by the embedding of "*" the start position. For pre- trained language model (PLM) with a dimension of d, we feed the document D to obtain the token embeddings Il and cross-token attention A:
where Hy is node representation .u denotes the set of edge types, W:O Rdxd and W Rdxd are learnable parameter matrices, and Ag is binary adjacency matrix for edges of type u.
We then conduct graph convolutional networks (GCN) (Kipf and Welling, 2017) over the dynamic document graph to model the interaction between nodes. The information transport is guided by the learned Á. This allows the information to be trans- ferred between nodes along important paths identi- fied by the learned weight Aij. Let - 9 denote the input and output representation of h GCN layer for node Ci. o can formally computed as follows:
H. A = PLM( (2.1111,2....... 4))
(I)
where H Reld and A Rixi, Both H and A take advantage of the last three transformer layers of the PLM.
The embedding for each entity e; is obtained by applying logsumexp pooling (Jia et al., 2019) over embeddings of corresponding mentions. Formally, for each entity E ED its representation he is computed as follows:
oe
A WoyWhoff gf ba) 95 (5)
where W and b are learnable parameter matrix and bias, is the Relu activation function.
[MMel
hr = log Σ exp(hm.)
(2)
After multiple convolutions on the document graph, we expect the mention nodes to have aggre- gated important information from relevant nodes. Similar to Eq. (2), we then apply logsumerp pool- ing on embeddings of mention nodes belonging to the same entity to obtain representations Jr. and for head and tail entities. To preserve the contextual information captured by the PLM, concatenate entity representations induced from the outputs of PLM with 9A or Sr and employ the bilinear func- tion to generate prediction probability of relation classification:
1=1
For each pair of entities (ch, et). the special con- textual representation Ch,t is obtained using the attention matrix A and token embeddings H.
Ch. = HT Ap & A
(3)
A% Ae
where A E Rt is attention to all the tokens for entity en, likewise for A.. In other words, both An, and A, are vectors that represent the relevance of entities' tokens to all the tokens in the document. @ denotes element- product.
2h = Lanh(WW lle..ellenon + bb)
(6)
4.4 Convolution over Dynamic Graph for Relation Extraction
tanh [he.helleneeelllggg +
(7)
We leverage the multi-head attention mechanism (Vaswani et al., 2017) dynamically learn the graph structure distinguish the importance of different edges. Specifically, the dynamic graph, containing n nodes, is represented by an n x n adjacency matrix with elements Agg in the range
o = W.W + by
(8)
where heg and he, are entity representations com- puted by Eq. (2), Ch,t are contextual representa- tion computed by Eq. (3), Wi, E Rexd W, E Raxd W,, E Rdxd are learnable paramters.

---

4.5 Evidence Supervision Module
are decreased below TH. This process enables the model effectively distinguish between positive and negative relations the input data.
the evidence supervision module, we introduce evidence retrieval as an auxiliary task and follow Ma et al. (2023) to utilize evidence distribution for enhancing the model's capability in filtering out irrelevant information. Specifically, for each entity pair (ce: 6a) with valid relation, weights all the tokens 9h Re is computed
We combine the loss of relation classification and evidence retrieval with a coefficient 3. The total training loss of our model can be formalized as:
L L + B x Levi-
(13)
A & A
Qh.t
(9) 4.7 Inference Stage Cross Fusion
A At
Inference Stage Fusion (ISF) (Xie et al., 2022) was proposed to leverage RE predictions from the original document and pseudo document D' constructed by evidence sentences. Two sets of prediction sores are merged through a blending layer (Wolpert, 1992) to obtain the final extraction results:
Then, for sentence 8; beginning from token indexed by m and ending at token indexed by its weight is obtained by adding up all weights of tokens within s;:
Σ j=m
The
(10)
Prusorlens (t) P(r|ch: Ct D)
Let Ph.t E RlSol be the importance distribu- tion for all sentences in the document. Then, we minimize the Kullback Leibler (KL) divergence between the extracted importance distribution Pl, and the evidence distribution Uh, E R!Spl derived from gold evidence labels:
(14)
+ P(rlen, 66 D')
where r is hyper-parameter tuned on development set.
ISF feeds the pseudo document into the model trained with evidence loss to obtain P(rle,, et; D The model trained with evidence loss naturally fo- cuses on selecting the parts interest from the input text, which is effective when the text contains a lot of irrelevant information. However, after the text has undergone one information filtering, i.e., evidence retrieval, the repeated information filter- ing process will lead to information loss, propose the Inference Stage Cross Fusion(ISCF)) which uses a model trained without evidence loss to perform inference on the pseudo document D'.
Levi
Σ light Ph.
Tha uhet
(11)
4.6 Training Objective
Since there may be multiple relations between two entities, formalize document- relation traction as a multi-label classification problem. Be- sides, a large portion of entity pairs have no valid relations. Following (Zhou et al., 2021), we intro- duce an adaptive threshold loss into our framework to address the relation imbalance issue. The loss of relation classification can be formalized as:
Experiments and Analysis
5.1
Datasets
Lre -- Σ log( exp(or)
DocRED (Yao al., 2019) is one of the most widely used datasets for document-level relation extraction. It contains 97 predefined relations, 63, 427 relational facts, and 5, 053 documents in total. However, Huang et al. (2022) have high- lighted the considerable noise introduced by the recommend-revis annotation scheme employed to construct DocRED. To address the problem of miss- ing labels within the DocRED, Tan et al. (2022b) proposed the Re-DocRED dataset which re-labels the DocRED dataset. Re-DocRED expands the quantity of relational facts in DocRED to a total of 119, 991, while providing clean dev and test
Er'E(P,,HH) )xp(o,)
rEPr
(12)
exp(orn)
- llog( E E(N,,T, exp(o:)
where P denotes positive classes and N denotes negative classes for an entity pair T (ch, et). TH denotes threshold relation to differentiate be- tween positive relation in P and negative relation in .. This is achieved by adjusting the logits of the positive and negative relations such that the logits positive relations are increased above the thresh- old value TH, while the logits negative relations

---

DocRED
Re-DoeRHD
DocRED test set were obtained by submitting the predictions CodaLab".
Datascts
Train
Dev
Train
Dev
Test
#Docs
3,053
1,000
3,053
500
500
Avg. #Anaphors
12.1
12.1
12.1
12.4
11.8
5.3
Main Results
Avg. #Mentions
21.2
21.3
21.2
21.3
21.4
Our experimental results DocRED are shown in Table 3, which indicates that our method consis- tently outperforms strong baselines and exist- ing SOTA model SAIS (Xiao et al., 2022a). Our BERT_basd base model exhibits notable enhancements in terms of F1 and Ign-F1, surpassing ATLOP- BERT_bace by 1.8 and 1.53 on the test set. These improvements highlight the effectiveness of explic- itly modeling the entity structure in the document. It is worth noting that our BERT_ base model im- proves on Fl and Ign-Fi by 1.86 and 1.84 over the previous graph-based based SOTA method GAIN- BERT base, demonstrating the importance of us- ing anaphors effectively to model cross-sentence entity interaction.
Avg. #Entities
19.5
19.6
19,4
19.4
19.6
Avg. #Triples
12.5
12.3
28.1
34.6
34.9
Avg. #Sentences
7.9
8.1
7.9
8.2
7.9
Table 1: Statistics of DocRED and Re-DocRED,
The statistics of DocRED and Re-DocRED datasets, including anaphors and mentions, are pre- sented Table 1.
sets.
5.2 Implementation Details
Our model is implemented using the PyTorch brary (Paszke et al., 2019) and HuggingFace Trans- formers (Wolf et al., 2019). All experiments are conducted on single NVIDIA A100 40GB GPU. We employ BERT base (Devlin et al., 2019) and RoBERTa_large (Liu al., 2019) for DocRED and RoBERTa_large for Re-DocRED document en- coders. Num of GCN layers, attention heads, and iterates were all set 2 in all experiments. All mod- els are trained with the AdamW optimizer (Kingma and Ba, 2015), accompanied by a warm-up sched- ule to facilitate the training process. All hyper- parameters are tuned based on the dev set. We list some of the parameterameters in Table 2.
Table 4 presents summary of the experimental results Re-DoeRED dataset. We observe pronounced performance gap between our model and baseline methods on Re-DocRED compared to the DocRED dataset. This disparity can at- tributed to the much cleaner data annotations in the Re-DocRED dataset, ensuring more fair basis for comparison. In Re-DocRED, our RoBERTa large- based model surpasses ATLOP with F1 and Ign-F1 scores of 3.47 and 3.18 higher, respectively.
5.4 Ablation Study
DocRED
Re-DocRED
Dataset
BERT
RoBERTa
RoBERTa
To examine the effectiveness of different compo- nents in our model, we conduct a series of ablation studies on both DocRED and Re- Re-DooRED, and the corresponding results are presented in Table 5. The detailed analysis outlined below:
epoch
30
30
30
rrcenddder
5e-5
3e-5
3e-5
Ir classiffier
le-4
lc-4
le-4
batch size
4
4
4
warmup ratio
6e-2
6e-2
6e-2
w/o Graph. Removing the dynamic graph leads to a degradation in model performance on both datasets, emphasizing the importance of explicitly modeling inner entity structure and cross-sentence entity interaction.
le-1
3e-2
5e-2
Table 2: Best hyper- parameerasss of our model observed on the development set.
w w/o ESM. We eliminate L_evi in the training pro- cess and adopt a typical fusion process. The Fi score shows decrease of 1.05, and 1.01 on the Do- cRED and Re-DocRED respectively. The decline in performance can be attributed to the factor of training without __evi which prevents the model from filtering out irrelevant information. w/o ISCF w ISF. By replacing the cross-fusion in the inference stage with a typical fusion process,
We employ F1, Ign-F1, Intra-F1, and Inter-FI to evaluate the performance of our model. Ign-F1 measures F1 by disregarding relation triples present in the training set. set. Intra-Fl evaluates F1 for relation triples that do not require inter-sentenee reasoning, whereas Inter-Fl evaluates F1 for relation triples that necessitate inter-sentence reasoning, To miti- gate potential bias, we present the average results of our model across 5 independent runs with cor- responding standard deviations. The results on the
Phttss:/codaaab.. www.:laabbiisuussssssse

---

Dev
Test
Model
PLM
Ign-P1
F1
Intra-F1
Inter-F1
Ign-F1
F1
LSR (Nan ul., 2020)
BERT buse
$2.43
59.00
65.26
$2.05
56.97
59.05
ATLOP (Zhou et al., 2021)
BERT base
59.11
61.011
67.26
53.20/
59.31
61.30
GAIN (Zeng et al., 2020)
BERT base
59.14
61.22
67.10
53.90
59.00
61.24
DocuNet (Zhang al., 2021)
BERT base
59.86
61.83
59.93
61.86
KD-DoeRE
al., 2022a)
BERT base
60.08
62.03
60.04
62.08
Fider (Xie et al., 2022)
BERT base
60.51
62.48
68.47
55.21
60.42
62.47
DREEAM (Ma et al., 2023)
BERT buse 60.51
62.55
60.03
62.49
SAIS (Xiao et al., 2022a)
BERT base
59.98
62.96
60.96
62.77
Ours
BERT
61.31:10.07
6..38:10.08
69.41:1.14
55.92:1.22
60.84
63.10
ATLOP (Zhou et al., 2021)
RoBERTa large
61.32
63.18
69.60
55.01
61.39
63.40
DocuNet (Zhang al., 2021)
RoBERTa large
62.23
64,12
62.39
64.55
KD-DocRR (Tan et al., 2022a)
RoBERTa, Larage
62.16
64.19
62.57
64,28
Eider (Xie ct al., 2022)
RoBERTa large
62.34
64.27
70.36
56.53
62.85
64.79
DREEAM (Ma et al., 2023)
RoBERTa large
62.29
64.20
62.12
64.27
SAIS (Xiao al., 2022a)
CoBERTa_large
62.23
65.17
63.44
65.11
Ours
RoBERTa large
63.15 + 0.05
65.19:10.09
71.09:10.08
57.83-10113
62.88
64.98
Table 3: Performance comparison between our approach and previous SOTA baseline methods on DocRED dataset. Results with are retrieved from Xie al. (2022).
Dev
Test
Model
Ign-Fl
F1
Ign-F1
F1
Intra-Fl
Inter-Fi F1
ATLOP (Zhou al., 2021)
76.88
77.63
76.94
77.73
80.18
75.13
DocuNet (Zhang et al., 2021)
77.53
78.16
77.27
77.92
79.91
76.64
KD-DocRE (Tan et al., 2022a)
77.92
78.65
77.63
78.35
79.57
77.26
DREEAM (Ma et al., 2023)
79.66
80.73
PEMSCL (Guo al., 2023)
79.02
79.89
79.01
79.86
Ours
80.04=0.10
81.15=0.12
80.12++0.07
81.20:+0.06
83,41+0.033
79.24:10.07
Table 4: Experimental results Re-DocRED dataset. Results of existing methods referred from Tan al. (2022b) and their corresponding original papers. The reported results are based on RoBERTa large.
Model
Ign-FI
Fl
Intra-Fl
Inter-FI
w/o Anaphor or Random replace. Removing replacing anaphors with randomly selected words resulted decrease in inter F1, while the intra Fl remained relatively consistent compared to sce- narios with anaphor retention. This supports our motivation to use anaphors for enhancing cross- sentence interactions between entities.
DocRED
Ours-BERT Mans-BRRT_baas base
61.33
63.38
69.30
56.03
w/o Graph
60.87
62.93
68.59
55.91
w/o ESM
60.06
62.33
68 42
54.81
ISCF or ISF
60.53
62.70
68.56
55.53
w/o Anaphor
61.07
63.02
69.32
55.21
Random replace
63.01
55.30
Re-DocRED
Burs-RoBERTTalarg
80.08
81.21
83.46
79.23
5.5
Case Study
wroo Graph
79.74
80.73
83.35
78.47
w/o ESM
79.12
80.20
82.71
77.98
In Figure 4, case study of our method is pre- sented, where we examine three sentences within a document containing mentions such as Carol II, Romania, Carol and Zizi Lambrino. Both ATLOP and DocuNet encounter difficulties in predicting the relationship of spouse between Carol II and Zizi Lambrino, primarily due to their failure in recog- nizing crucial anaphors, such he, his. In contrast, our model adeptly captures anaphora information within the sentences, facilitating more compre- hensive understanding of the interaction between Carol II and Zizi Lambrino, ultimately leading to
w/o ISCF w ISF
79.03
80.18
82.53
78.11
w/o Anaphor
79.91
81.00
83.17
79.08
Random replace
79.70
80.89
83.47
78.67
Table 5: Ablation study on DocRED dev set and Re- DocRED test We DocRED dev due the testing set of DocRED is not publicly available.
we observe a decrease in all metrics on two datasets. These results suggest that the duplicate information filtering process may not be justified to properly utilize pseudo-dos docummntss.

---

[0] Carol If 15 October 18934 April 1953 reigned King of Romania from 8 June 1930 until his coerced... [1] Carol was eldest son Ferdinand became crown prince upon the death of grand uncle, King Carol I in 1914.
[4] He possessed a hedonistic personality that contributed to the controversies marrigg his reign, und his life was marked mumerous scandals, among them marriages Zizi Lambrino and....
Zisi Lambrino
Zizi Lambrino
state
Cama II
Romania
Carai If
Romania
country of citizenship
Country of chrizessip
country of citizanship
country of citizensiip
ATLOP
Carol
DocuNet
Caroi
spruss
Zizi Lambrino
Spease spouse
Zizi Lumbrino
spouse
bead of state
head of state
Carri
Romania
Carol II
Romania
country of citizenship
country of eirccnssip
head of stare
countay citizeneiig
country of crizenship
head of state
Carol I
Carol
Ours
Label
Figure 4: A case study of our method. Only a part of entities and sentences are displayed due to space limitation.
the successful identification of their relationship.
trade-off between evidence retrieval loss and rela- tion classification loss. In Figure 6, the value of increases, the Fl score shows an overall trend of initially ascending and then subsequently declin- ing. This pattern suggests that the optimal balance between L.e. and Levi tends to fall within the range of O to 0.1. It is evident that the choice of 3 greatly affects the effectiveness and efficacy of the RE task at the document level.
5.6 Impact of GCN Layers
We conducted experiments on DocRED based on BERT base without fusion to analyze the poten- tial influence of the number of GCN layers. As depicted in Figure 5, there is significant increase in the Intra-Fi score when transitioning from 0 to GCN layer. This can be attributed to the rea- son that GCN facilitates efficient information trans- fer among different nodes such as m, < il and m, + Mj. But single layer of GCN is still insufficient to explicitly capture cross-sentence en- tity interactions such as m, the show -> mj m; it + mj- The Inter-F1 score increase when the number of GCN layers is set to Two- layer GCNs can enhance cross-sentence entity in- teractions by effectively aggregating information through anaphors.
(a) F1
(b) Inter-F1
(c) Intra-F1
Figure 6: Impact of evidence assistance coefficient 8.
Conclusions
In this paper. we present an approach that explic- itly and jointly models coreference and anaphora information to effectively capture entities' inter- nal structure and external interactions. Moreover, we employ dynamic algorithm for graph pruning and structural optimization, which requires min- imal additional annotations. We also introduce evidence retrieval auxiliary task enhance the encoder. Empirical studies conducted on well- established benchmarks confirm the effectiveness of our proposed model.
(a) Fl
(b) Inter-F1
(c) Intra-Fi
Figure Impact of GCN layers K.
5.7
Impact of Coefficient 3
our proposed method, the evidence assistance coefficient 3 plays a crucial role in regulating the

---

Acknowledgements
2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume 1 (Long and Short Papers), pages 3693-3704, Minneapolis, Min- nesota. Association for Computational Linguistics.
This work is supported partly by the Na- tional Key R&D Program of China under Grant 2021ZD01 10700, partly by the Fundamental Re- search Funds for the Central Universities, and partly by the State Key Laboratory of Software Development Environment.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations. ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Limitations
Our method has certain limitations that should be acknowledged. Firstly, the anaphors used in our method are acquired by an external parser, which has risk of introducing potential errors caused by the external parser. Secondly, our model's gen- eralization may be insufficient in cases where the document contains limited number of anaphors.
Thomas N. Kipf and Max Welling. 2017. Semi- supervised classification with graph convolutional networks. In Sth International Conference on Learn- ing Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Bo Li, Wei Ye, Zhonghao Sheng. Rui Xie, Xiangyu Xi, and Shikun Zhang. 2020. Graph enhanced dual attention network for document-leves relation extrac- tion. Proceedings of the 28th International Confer- ence on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 1551-1560. International Committee on Com- putational Linguistics.
References
Fenia Christopoulou, Makoto Miwa, and Sophia Ana- niadou. 2019. Connecting the dots: Document-level neural relation extraction with edge-oriented graphs. In Proceedings of the 2019 Conference on Empiri- cal Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. JAMNLP-JJCNL 2019, Hong Kong, China, November 3-7, 2019, pages 4924 4935. Association for Computational Linguistics.
Yinhan Liu, Myle Ott. Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv:
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and
Youmi Ma, An Wang, and Naoaki Okazaki. 2023. DREEAM: guiding attention with evidence for im- proving document-level relation extraction. In Pro- ceedings of the 17th Conference of the European Chapter of the Association for Computational Lin- guistics, EACII 2023, Dubrovnik, Croatia, May 2-6, 2023, pages 1963-1975. Association for Computa- tional Linguistics.
Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Guoshunn Nan, Zhijiang Guo, Ivan Sekulic, and Wei Lu. 2020. Reasoning with latent structure refinement for document- level relation extraction. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 1546-1557, Association for Computational Linguistics.
Jia Guo, Stanley Kok, and Lidong Bing. 2023. To- wards integration of discriminability and robustness for document-level relation extraction. In Proceed- ings of the 17th Conference of the European Chap- ter of the Association for Computational Linguistics, EACL 2023, Dubrovnik, Croatia, May 2-6, 2023, pages 2598-2609. Association for Computational Linguistics.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alhan Desmaison, Andreas Köpf, Edward 7. Yang. Zachary De Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy. Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style. high-performant deep learning li- brary. In Advances in Neural Information Processing Systems 32: Anmual Conference on Neural Informa- tion Processing Systems 2019, NewrIPS 2019, De- cember 8-14, 2019, Vancouver, BC, Canada, pages 8035-
Quzhe Huang, Shibo Hao, Yuan Ye, Shengqi Zhu, Yansong Feng, and Dongyan Zhao. 2022. Does recommend-rew revise produce reliable annotations? an analysis on missing instances in DocRED. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume I: Long Papers), pages 6241-6252, Dublin, Ireland. Associa- tion for Computational Linguisties.
Robin Jia, Cliff Wong, and Hoifung Poon. 2019. Document-level n-ary relation extraction with multi- scale representation learning. In Proceedings of the

---

Sumil Kumar Sahu, Fenia Christopoulou, Makoto Miwa, and Sophia Ananiadou. 2019. Inter-sentence relation extraction with document-level graph convolutional neural network. In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, pages 1309-4316, Florence, Italy, Associa- tion for Computational Linguisties.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019, Huggingface's 's transformers: Statc-of- the-art natural language processing. arXiv preprint arXiv: 1910.0771..
David H Wolpert. 1992. Stacked generalization. Neural networks, 5(2):241-259
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. 2008. The graph neural network model. IEEE Iransactions on neural networks, 20(1):61-88 600
Yuxin Xiao, Zecheng Zhang, Yuning Mao, Carl Yang. and Jiawei Han. 2022a. SAIS: supervising and aug- menting intermediate steps for document-level re- lation extraction. In Proceedings of the 2022 Con- ference of the North American Chapler of the As- sociation for Computational Linguistiex: Human Language Technologies, NAACL 2022, Seattle, WA, United States. July 10-15, 2022, pages 2395-2409. Association for Computational Linguistics.
Kai Sun, Richong Zhang, Samuel Mensah, Yongyi Mao, and Xudong Liu. 2020. Recurrent interaction net- work for jointly extracting entities and classifying relations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3722-3732, Online, Association for Computational Linguistics.
Yuxin Xiao, Zecheng Zhang. Yuning Mao, Carl Yang. and Jiawei Han. 2022b. SAIS: Supervising and aug- menting intermediate steps for document-level re- lation extraction. In Proceedings of the 2022 Con- ference of the North American Chapter of the As- sociation for Computational Linguisties: Human Language Technologies, pages 2395-2409, Seatle,, United States. Association for Computational Lin- guistics.
Qingyu Tan, Ruidan He, Lidong Bing, and Hwee Tou Ng. 2022a. Document-lews relation extraction with adaptive focal loss and knowledge distillation. In Findings of the Association for Computational Lin- guisticy; ACL 2022, pages 1672-1681, Dublin, Ire- land. Association for Computational Linguistics.
Qingyu Tan, Lu Xu, Lidong Bing. Hwee Tou Ng, and Sharifah Mahani Aljunied. 2022b. Revisiting docred addressing the false negative problem in relation extraction. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7- 2022, pages 2ages8472-285 8472-8487 Association for Computational Linguistics.
Yiqing Xie, Jiaming Shen, Sha Li, Yuning Mao, and Ji- awei Han. 2022. Fider: Empowering document-level relation extraction with efficient evidence extraction and inference-stage fusion. In Findings of the As- sociation for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 257-268. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Nefural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Wang Xu, Kehai Chen, Lili Mou, and Tiejun Zhao. 2022. Document-level relation extraction with sentences importance estimation and focusing. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 2920- 2929, Seattle, United States. Association for Computational Linguistics.
Difeng Wang, Wei Hu, Ermei Cao, and Weijian Sun. 2020a. Global-to-local neural networks for document-less level relation extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 371 1- 3721, Online. Association for Computational Lin- guistics.
Wang Xu, Kehai Chen, and Tiejun Zhao. 2021. Document-level relation extraction with reconstruc- tion. In Proceedings of the AAAI Conference Arti- ficial Intelligence, volume 35, pages 14167-
Difeng Wang, Wei Hu, Frmei Cao, and Weijian Sun. 2020b. Global-to- loccall neural networks for document-level relation extraction. In Proceedings of the 2020 Conference on Empirical Methods in Nut- ural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 3711-3721. Associa- tion for Computational Linguistics.
Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang. Jie Zhou, and Maosong Sun. 2019. DocRED: A large-scale document-lever relation extraction dataset. In Pro- ceedings of the 57th Annual Meeting of the Associa- tion for Computational Linguistics, pages 764-777, Florence, Italy, Association for Computational Lin- guistics.
Peifeng Wang, Nanyun Peng, Filip Ilievski, Pedro Szekely, and Xiang Ren. 2020c. Connecting the dots: knowledgeable path generator for commonsense question answering. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages Linguistics.
Jiaxin Yu. Deqing Yang, and Shuyu Tian. 2022. Relation-specific attentions over entity mentions for enhanced document-level relation extraction. In Pro- ceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational
Online. Association for Computational

---

Linguistics: Human Language Technologies. pages 1529, Seattle, United States. Association for Computational Linguistics.
Shuang Zeng, Runxin Xu, Baobao Chang, and Lei Li. 2020. Double graph based reasoning for document- level relation extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages (630-1640, On- line. Association for Computational Linguistics.
Ningyu Zhang, Xiang Chen, Xin Xie, Shumin Deng, Chuanqi Tan, Mosha Chen, Fei Huang, Luo Si, and Huajun Chen. 2021. Document-level relation extrac- tion as semantic segmentation. In Proceedings of the Thirtieth International Joint Conference Artificial Intelligence, IJCAI 2021, Virtual Event Montreal, Canada, 19-27 August 2021, pages 3999-4006.. ij- cai.org.
Yuhao Zhang, Peng Qi. and Christopher D. Manning. 2018. Graph convolution over pruned dependency trees improves relation extraction. In Proceedings of the 2018 Conference Empirical Methods in Nat- ural Language Processing, pages 2205-2215, Brus- sels, Belgium. Association for Computational Lin- guistics.
Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher Manning. 2017. Position-awar attention and supervised data improve slot filling, In Proceedings the 2017 Conference Empiri- Methods in Natural Language Processing, pages 35-45, Copenhagen, Denmark. Association for Com- putational Linguisties.
Yanpeng Zhao, Jack Hessel, Youngjae Yu, Ximing Lu, Rowan Zellers, Yejin Choi. 2022. Connecting the dots between audio and text without parallel data through visual knowledge transfer. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 4492-4507, Seatlle, United States. Association Computational Linguistics.
Wenxuan Zhou, Kevin Huang, Tengyu Ma, and Jing Huang. 2021. Document-leves relation extraction with adaptive thresholding and localized context pool- ing. In Proceedings of the AAAI conference on artifi- cial intelligence, volume 35, pages 14612-14620
Hao Zhu, Yankai Lin, Zhiyuan Liu, Jie Fu, Tat-Seng Chua, and Maosong Sun. 2019. Graph neural net- works with generated parameters for relation extrac- tion. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1331-1339, Florence, Italy. Association for Compu- tational I Linguistics.


Reviews:
[{"review_id": "yb810fAuNS", "reviewer": "Reviewer_dWHw", "summary": "", "questions": "1. May I know how are the anaphor nodes obtained? On average, how many anaphors and mentions are there in a document?\n2. Besides, suppose the document has M mentions and N anaphors on average, does that mean there are M*N Mention-Anaphora Edges? or there are pruning mechanisms for such edges?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Questions For The Authors:**\n\nR1:\n\nThank you for bringing this to our attention. Regarding the details on obtaining the anaphor nodes, we wish to provide a detailed description here.\n\nOur approach to obtaining anaphor nodes involves two distinct rules (Lines 443 to 446):\n\n1. We utilize Spacy's tagger functionality to perform parts-of-speech tagging, treating all pronouns (PRON) as potential anaphors, as they may refer back to a mention.\n2. We also leverage Spacy's dependency parser to help identify anaphors. Specifically, in instances where a token's dependency relation is identified as 'det' (i.e., determiner) and its text is 'the', the approach involves identifying all tokens positioned between this particular 'the' token and its associated head as an anaphor. For instance, if the associated head of the ‘the’ token happens to be the word ‘song’ in the text ‘The song is a rock with elements of alternative rock.’, the resulting anaphor extracted is ‘the song’.\n\nAddressing your inquiry regarding the number of anaphors and mentions in a document on average, we provide the statistics here:\n\n|                | DocRED_train | DocRED_**dev** | DocRED_**test** | **ReDocRED_train** | **ReDocRED_dev** | ReDocRED_**test** |\n| -------------- | ------------ | -------------- | --------------- | ------------------ | ---------------- | ----------------- |\n| #Anaphors      | 36990        | 12088          | 11964           | 36990              | 6189             | 5899              |\n| Avg. #Anaphors | 12.1         | 12.1           | 12.0            | 12.1               | 12.4             | 11.8              |\n| #Mentions      | 64876        | 21336          | 21468           | 64876              | 10652            | 10684             |\n| Avg. #Mentions | 21.2         | 21.3           | 21.5            | 21.2               | 21.3             | 21.4              |\n\nWe will include these additional details to further strengthen our work.\n\n\n\nR2:\n\nIndeed, if a document holds an average of M mentions and N anaphors, it implies the presence of M*N Mention-Anaphora Edges (Lines 247 to 255) within our constructed graph. Our model depends on an attention mechanism (Eq.(4))  as a soft pruning mechanism to dynamically select important Mention-Anaphora Edges in the graph.\n\n\n\n**Typos Grammar Style And Presentation Improvements:**\n\nR1:\n\nThank you for your suggestion regarding further details about the method used to obtain Anaphora Nodes. In the revised version of our paper, we will provide additional details to ensure a clearer understanding of this aspect.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "To the best of my knowledge, this is the first work to integrate pronoun-level coreference information in document-level RE, which is important for cross-sentence relation discovery. This work also leverages GNNs to learn Mention-Anaphora edges, which require minimal extra annotations. This framework improves the performances on DocRED and Re-DocRED datasets.", "reasons_to_reject": "The presentation could be better. GNN systems are already proposed in document-level RE. In my opinion, the main contribution of the paper is introducing anaphor nodes and edges. However, the details on how to obtain such anaphors are not provided in section 4.1.", "questions_to_authors": "1. May I know how are the anaphor nodes obtained? On average, how many anaphors and mentions are there in a document?\n2. Besides, suppose the document has M mentions and N anaphors on average, does that mean there are M*N Mention-Anaphora Edges? or there are pruning mechanisms for such edges?", "paper_topic_and_main_contributions": "This work prosed an anaphor-assisted framework for document-level relation extraction. This framework leverages pronouns to capture the fine-grained interactions between entities.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "It will be better if the authors could provide more details on the method that they used to obtain the Anaphora Nodes. The concept of Anaphora Nodes is important."}, "extra_text": {}}, {"review_id": "KyNLrFv3A9", "reviewer": "Reviewer_VsBi", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 1, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Reasons To Reject:**\n\nR1:\n\nWe appreciate your feedback regarding the introduction section. Following your suggestion, we will provide a summary of our contributions to ensure that readers can quickly grasp the novel aspects of the paper.\n\n\n\nR2:\n\nIn our paper, we have incorporated a multi-task learning setup, which involves optimizing multiple loss functions simultaneously (Eq.(13)). The evidence assistance coefficient β enables us to regulate the emphasis placed on learning from the two tasks simultaneously. In Figure 5, as the value of β increases, the F1 score shows an overall trend of initially ascending and then subsequently declining. This pattern suggests that the optimal balance between loss_re and loss_evi tends to fall within the range of 0 to 0.1.\n\nIn the revised version, we will provide a more detailed explanation to highlight how varying the value of β leads to changes in the F1 score, with an effort to improve the clarity of this experiment.\n\n**Typos Grammar Style And Presentation Improvements:**\n\nR1:\n\nThank you for pointing out the equation reference error on line 334. We apologize for this oversight and will make the necessary corrections in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The experiment is relatively detailed, and the logic of the article is relatively easy to understand.\n2. Experimental results on the widely-used datasets demonstrate that the model achieves a new state-of-the-art performance.", "reasons_to_reject": " 1. In the introduction section, there is a lack of a summary regarding the innovations of this paper.\n2. In Figure 5, it is not clear how the value of the evidence assistance coefficient β affects the model performance. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper, the authors proposed an anaphor-assisted framework for DocRE tasks.  In order to explicitly and jointly leverage the co-reference and anaphora information in the document, the authors constructed a  document graph by considering all possible entity mentions and anaphors. They first employed an attention-based graph convolutional neural network to learn the structure of the graph dynamically. Additionally, they introduced evidence verification as an auxiliary task to assist the model in filtering out irrelevant information. ", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. The equation reference in the paper is incorrect on line 334. It should be \"Similar to Eq.(2)\" instead of \"Eq.(4.2)."}, "extra_text": {}}, {"review_id": "CVPE0J6IkJ", "reviewer": "Reviewer_cmU3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Reasons To Reject:**\n\nR1:\n\nThank you for your comment. In our view, we believe the utilization of multiple testing techniques serves to validate the robustness and effectiveness of our proposed approach. Regarding the lack of highlighting the proposed method, we intend to address this concern by including a summary in the introduction to highlight the contributions of our work, allowing readers to quickly grasp the novel aspects of the paper.\n\n\n\nR2:\n\nWe appreciate the suggestion regarding the need for an analysis of the proposed graph structure in relation to the anaphora. In response, we have conducted two distinct experiments. The first involves removing anaphora nodes from the document graph (w/o anaphora), while the second replaces anaphora nodes with randomly selected words from the sentences (Random replace). The results are presented as following:\n\n| **Model**              | **Ign-F1** | **F1** | **Intra-F1** | **Inter-F1** |\n| ---------------------- | ---------- | ------ | ------------ | ------------ |\n| DocRED_BERT_base       | 61.33      | 63.38  | 69.30        | 56.03        |\n| w/o anaphor            | 61.07      | 63.02  | 69.32        | 55.21        |\n| Random replace         | 60.96      | 63.01  | 69.28        | 55.30        |\n| ReDocRED_RoBERTa_large | 80.08      | 81.21  | 83.46        | 79.23        |\n| w/o anaphor            | 79.91      | 81.00  | 83.17        | 79.08        |\n| Random replace         | 79.70      | 80.89  | 83.47        | 78.67        |\n\nThe key insight drawn from this experiment is that the results on both the DocRED and ReDocRED datasets highlight the importance of retaining anaphora nodes in the proposed graph structure. We will include these results in our revised version.\n\n\n\nR3:\n\nYour suggestion to include an ablation study involving the anaphora node is insightful, and we will include the ablated results of removing anaphora nodes in the revised version.\n\nAs the results in the table shown in R2, the removal of anaphora nodes was observed to lead to a decrease in inter F1, while the intra F1 remained relatively consistent compared to scenarios where anaphora nodes were retained. This aligns well with our motivation, which centers on utilizing anaphora to enhance cross-sentence interactions between entities.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed methods exhibit commendable performance.\n2. The paper is lucid and skillfully composed.", "reasons_to_reject": "1. The paper utilise mutlple teisting techniques and does not highlight the proposed methods.\n2. The disparity between the motavtion and the experiments analysis. The paper proposed to explicitly and jointly model coreference and anaphora information and utilize a dynamic algorithm. However the experiments did not focus on the analysis of the proposed graph structure related to the anaphora, rather than the existing techniques.\n3. The experiments are not completion. To solve the problem of anaphora, the paper proposes to introduce the anaphora node. To show the effective of the graph structure, the ablation of the anaphora node should be include.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper highlights that inadequate preprocessing of the anaphor has hindered the ability to comprehend intricate interactions between entities. In response, the authors propose an anaphor-assisted framework for the DocRE task. The experiments demonstrate the efficacy of the proposed approach.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is well-written, and propose an innovative method for document-level relation extraction. Presentation should be improved according to reviewers’ comments.
Decision: Accept-Main