SlowGuess/ACL_2020_OCR huggingface dataset:
Guoshun  $\mathsf{Nan}^{1*}$ , Zhijiang Guo $^{1*}$ , Ivan Sekulic $^{2\dagger}$  and Wei Lu $^{1}$
$^{1}$ StatNLP Research Group, Singapore University of Technology and Design
<sup>2</sup>Università della Svizzera italiana
guoshun_nan@sutd.edu.sg, zhijiang_quo@mymail.sutd.edu.sg ivan.sekulic@usi.ch, luwei@sutd.edu.sg
Document-level relation extraction requires integrating information within and across multiple sentences of a document and capturing complex interactions between inter-sentence entities. However, effective aggregation of relevant information in the document remains a challenging research question. Existing approaches construct static document-level graphs based on syntactic trees, co-references or heuristics from the unstructured text to model the dependencies. Unlike previous methods that may not be able to capture rich non-local interactions for inference, we propose a novel model that empowers the relational reasoning across sentences by automatically inducing the latent document-level graph. We further develop a refinement strategy, which enables the model to incrementally aggregate relevant information for multi-hop reasoning. Specifically, our model achieves an  $F1$  score of 59.05 on a large-scale document-level dataset (DocRED), significantly improving over the previous results, and also yields new state-of-the-art results on the CDR and GDA dataset. Furthermore, extensive analyses show that the model is able to discover more accurate inter-sentence relations.
Relation extraction aims to detect relations among entities in the text and plays a significant role in a variety of natural language processing applications. Early research efforts focus on predicting relations between entities within the sentence (Zeng et al., 2014; Xu et al., 2015a,b). However, valuable relational information between entities, such as biomedical findings, is expressed by multiple mentions across sentence boundaries in real-world scenarios (Peng et al., 2017). Therefore, the scope
Figure 1: An example adapted from the DocRED dataset. The example has four entities: Lutsenko, internal affairs, Yulia Tymoshenko and Ukrainian. Here entity Lutsenko has two mentions: Lutsenko and He. Mentions corresponding to the same entity are highlighted with the same color. The solid and dotted lines represent intra- and inter-sentence relations, respectively.
of extraction in biomedical domain has recently been expanded to cross-sentence level (Quirk and Poon, 2017; Gupta et al., 2018; Song et al., 2019).
A more challenging, yet practical extension, is the document-level relation extraction, where a system needs to comprehend multiple sentences to infer the relations among entities by synthesizing relevant information from the entire document (Jia et al., 2019; Yao et al., 2019). Figure 1 shows an example adapted from the recently proposed document-level dataset DocRED (Yao et al., 2019). In order to infer the inter-sentence relation (i.e., country of citizenship) between Yulia Tymoshenko and Ukrainian, one first has to identify the fact that Lutsenko works with Yulia Tymoshenko. Next we identify that Lutsenko manages internal affairs, which is a Ukrainian authority. After incrementally connecting the evidence in the document and performing the step-by-step reasoning, we are able to infer that Yulia Tymoshenko is also a Ukrainian.
Prior efforts show that interactions between mentions of entities facilitate the reasoning process in the document-level relation extraction. Thus, Verga et al. (2018) and Jia et al. (2019) leverage Multi-Instance Learning (Riedel et al., 2010; Surdeanu
* Equally Contributed.
† Work done during internship at SUTD.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1546-1557 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
et al., 2012). On the other hand, structural information has been used to perform better reasoning since it models the non-local dependencies that are obscure from the surface form alone. Peng et al. (2017) construct dependency graph to capture interactions among  $n$ -ary entities for cross-sentence extraction. Sahu et al. (2019) extend this approach by using co-reference links to connect dependency trees of sentences to construct the document-level graph. Instead, Christopoulou et al. (2019) construct a heterogeneous graph based on a set of heuristics, and then apply an edge-oriented model (Christopoulou et al., 2018) to perform inference.
Unlike previous methods, where a document-level structure is constructed by co-references and rules, our proposed model treats the graph structure as a latent variable and induces it in an end-to-end fashion. Our model is built based on the structured attention (Kim et al., 2017; Liu and Lapata, 2018). Using a variant of Matrix-Tree Theorem (Tutte, 1984; Koo et al., 2007), our model is able to generate task-specific dependency structures for capturing non-local interactions between entities. We further develop an iterative refinement strategy, which enables our model to dynamically build the latent structure based on the last iteration, allowing the model to incrementally capture the complex interactions for better multi-hop reasoning (Welbl et al., 2018).
Experiments show that our model significantly outperforms the existing approaches on DocRED, a large-scale document-level relation extraction dataset with a large number of entities and relations, and also yields new state-of-the-art results on two popular document-level relation extraction datasets in the biomedical domain. The code and pretrained model are available at https://github.com/nanguoshun/LSR<sup>1</sup>.
- We construct a document-level graph for inference in an end-to-end fashion without relying on co-references or rules, which may not always yield optimal structures. With the iterative refinement strategy, our model is able to dynamically construct a latent structure for improved information aggregation in the entire document.
- We perform quantitative and qualitative analyses to compare with the state-of-the-art mod
els in various settings. We demonstrate that our model is capable of discovering more accurate inter-sentence relations by utilizing a multi-hop reasoning module.
In this section, we present our proposed Latent Structure Refinement (LSR) model for the document-level relation extraction task. Our LSR model consists of three components: node constructor, dynamic reasoner, and classifier. The node constructor first encodes each sentence of an input document and outputs contextual representations. Representations that correspond to mentions and tokens on the shortest dependency path in a sentence are extracted as nodes. The dynamic reasoner is then applied to induce a document-level structure based on the extracted nodes. Representations of nodes are updated based on information propagation on the latent structure, which is iteratively refined. Final representations of nodes are used to calculate classification scores by the classifier.
2.1 Node Constructor
Node constructor encodes sentences in a document into contextual representations and constructs representations of mention nodes, entity nodes and meta dependency paths (MDP) nodes, as shown in Figure 2. Here MDP indicates a set of shortest dependency paths for all mentions in a sentence, and tokens in the MDP are extracted as MDP nodes.
2.1.1 Context Encoding
Given a document  $d$ , each sentence  $d_{i}$  in it is fed to the context encoder, which outputs the contextualized representations of each word in  $d_{i}$ . The context encoder can be a bidirectional LSTM (BiLSTM) (Schuster and Paliwal, 1997) or BERT (Devlin et al., 2019). Here we use the BiLSTM as an example:
$$
\overleftarrow {\mathbf {h} _ {j} ^ {i}} = \mathbf {L S T M} _ {l} \left(\overleftarrow {\mathbf {h} _ {j + 1} ^ {i}}, \gamma_ {j} ^ {i}\right) \tag {1}
$$
$$
\overrightarrow {\mathbf {h} _ {j} ^ {i}} = \mathbf {L S T M} _ {r} \left(\overrightarrow {\mathbf {h} _ {j - 1} ^ {i}}, \gamma_ {j} ^ {i}\right) \tag {2}
$$
where  $\overleftarrow{\mathbf{h}}_j^i,\overleftarrow{\mathbf{h}}_{j + 1}^i,\overrightarrow{\mathbf{h}}_j^i$  and  $\vec{\mathbf{h}}_j^i$  represent the hidden representations of the  $j$ -th,  $(j + 1)$ -th and  $(j - 1)$ -th token in the sentence  $d_{i}$  of two directions, and  $\gamma_j^i$  denotes the word embedding of the  $j$ -th token. Contextual representation of each token in the sentence is represented as  $\mathbf{h}_j^i = [\overleftarrow{\mathbf{h}}_j^i;\overrightarrow{\mathbf{h}}_j^i]$  by concatenating hidden states of two directions, where  $\mathbf{h}_j^i\in \mathbb{R}^d$  and  $d$  is the dimension.
<sup>1</sup>Our model is implemented in PyTorch (Paszke et al., 2017)
Figure 2: Overview of the Node Constructor: A context encoder is applied to get the contextualized representations of sentences. The representations of mentions and words in the meta dependency paths are extracted as mention nodes and MDP nodes. An average pooling is used to construct the entity node from the mention nodes. For example, the entity node Lutsenko is constructed by averaging representations of its mentions Lutsenko and He. All figures best viewed in color.
2.1.2 Node Extraction
We construct three types of nodes for a document-level graph: mention nodes, entity nodes and meta dependency paths (MDP) nodes as shown in Figure 2. Mention nodes correspond to different mentions of entities in each sentence. The representation of an entity node is computed as the average of its mentions. To build a document-level graph, existing approaches use all nodes in the dependency tree of a sentence (Sahu et al., 2019) or one sentence-level node by averaging all token representations of the sentence (Christopoulou et al., 2019). Alternatively, we use tokens on the shortest dependency path between mentions in the sentence. The shortest dependency path has been widely used in the sentence-level relation extraction as it is able to effectively make use of relevant information while ignoring irrelevant information (Bunescu and Mooney, 2005; Xu et al., 2015a,b). Unlike sentence-level extraction, where each sentence only has two entities, each sentence here may involve multiple mentions.
2.2 Dynamic Reasoner
The dynamic reasoner has two modules, structure induction and multi-hop reasoning as shown in Figure 3. The structure induction module is used to learn a latent structure of a document-level graph. The multi-hop reasoning module is used to perform inference on the induced latent structure, where representations of each node will be updated based on the information aggregation scheme. We stack  $N$  blocks in order to iteratively refine the latent document-level graph for better reasoning.
2.2.1 Structure Induction
Unlike existing models that use co-reference links (Sahu et al., 2019) or heuristics (Christopoulou et al., 2019) to construct a document-level graph
Figure 3: Overview of the Dynamic Reasoner. Each block consists of two sub-modules: structure induction and multi-hop reasoning. The first module takes the nodes constructed by the Node Constructor as inputs. Representations of nodes are fed into two feed-forward networks before the bilinear transformation. The latent document-level structure is computed by the Matrix-Tree Theorem. The second module takes the structure as input and updates representations of nodes by using the densely connected graph convolutional networks. We stack  $N$  blocks which correspond to  $N$  times of refinement. Each iteration outputs the latent structure for inference.
for reasoning, our model treats the graph as a latent variable and induces it in an end-to-end fashion. The structure induction module is built based on the structured attention (Kim et al., 2017; Liu and Lapata, 2018). Inspired by Liu and Lapata (2018), we use a variant of Kirchhoff's Matrix-Tree Theorem (Tutte, 1984; Koo et al., 2007) to induce the latent dependency structure.
Let  $\mathbf{u}_i$  denote the contextual representation of the  $i$ -th node, where  $\mathbf{u}_i \in \mathbb{R}^d$ , we first calculate the pair-wise unnormalized attention score  $\mathbf{s}_{ij}$  between the  $i$ -th and the  $j$ -th node with the node represent
tations  $\mathbf{u}_i$  and  $\mathbf{u}_j$ . The score  $\mathbf{s}_{ij}$  is calculated by two feed-forward neural networks and a bilinear transformation:
$$
\mathbf {s} _ {i j} = (\tanh (\mathbf {W} _ {p} \mathbf {u} _ {i})) ^ {T} \mathbf {W} _ {b} (\tanh (\mathbf {W} _ {c} \mathbf {u} _ {j})) (3)
$$
where  $\mathbf{W}_p\in \mathbb{R}^{d\times d}$  and  $\mathbf{W}_c\in \mathbb{R}^{d\times d}$  are weights for two feed-forward neural networks,  $d$  is the dimension of the node representations, and tanh is applied as the activation function.  $\mathbf{W}_b\in \mathbb{R}^{d\times d}$  are the weights for the bilinear transformation. Next we compute the root score  $\mathbf{s}_i^r$  which represents the unnormalized probability of the  $i$ -th node to be selected as the root node of the structure:
$$
\mathbf {s} _ {i} ^ {r} = \mathbf {W} _ {r} \mathbf {u} _ {i} \qquad (4)
$$
where  $\mathbf{W}_r\in \mathbb{R}^{1\times d}$  is the weight for the linear transformation. Following Koo et al. (2007), we calculate the marginal probability of each dependency edge of the document-level graph. For a graph  $\mathbf{G}$  with  $n$  nodes, we first assign non-negative weights  $\mathbf{P}\in \mathbb{R}^{n\times n}$  to the edges of the graph:
$$
\mathbf {P} _ {i j} = \left\{ \begin{array}{l l} 0 & \text {i f} i = j \\ \exp \left(\mathbf {s} _ {i j}\right) & \text {o t h e r w i s e} \end{array} \right. \tag {5}
$$
where  $\mathbf{P}_{ij}$  is the weight of the edge between the  $i$ -th and the  $j$ -th node. We then define the Laplacian matrix  $\mathbf{L} \in \mathbb{R}^{n \times n}$  of  $\mathbf{G}$  in Equation (6), and its variant  $\hat{\mathbf{L}} \in \mathbb{R}^{n \times n}$  in Equation (7) for further computations (Koo et al., 2007).
$$
\mathbf {L} _ {i j} = \left\{ \begin{array}{l l} \sum_ {i ^ {\prime} = 1} ^ {n} \mathbf {P} _ {i ^ {\prime} j} & \text {i f} i = j \\ - \mathbf {P} _ {i j} & \text {o t h e r w i s e} \end{array} \right. \tag {6}
$$
$$
\hat {\mathbf {L}} _ {i j} = \left\{ \begin{array}{l l} \exp \left(\mathbf {s} _ {i} ^ {r}\right) & \text {i f} i = 1 \\ \mathbf {L} _ {i j} & \text {i f} i > 1 \end{array} \right. \tag {7}
$$
We use  $\mathbf{A}_{ij}$  to denote the marginal probability of the dependency edge between the  $i$ -th and the  $j$ -th node. Then,  $\mathbf{A}_{ij}$  can be derived based on Equation (8), where  $\delta$  is the Kronecker delta (Koo et al., 2007).
$$
\begin{array}{l} \mathbf {A} _ {i j} = (1 - \delta_ {1, j}) \mathbf {P} _ {i j} [ \hat {\mathbf {L}} ^ {- 1} ] _ {i j} \tag {8} \\ - (1 - \delta_ {i, 1}) \mathbf {P} _ {i j} [ \hat {\mathbf {L}} ^ {- 1} ] _ {j i} \\ \end{array}
$$
Here,  $\mathbf{A} \in \mathbb{R}^{n \times n}$  can be interpreted as a weighted adjacency matrix of the document-level entity graph. Finally, we can feed  $\mathbf{A} \in \mathbb{R}^{n \times n}$  into the multi-hop reasoning module to update the representations of nodes in the latent structure.
2.2.2 Multi-hop Reasoning
Graph neural networks have been widely used in different tasks to perform multi-hop reasoning (Song et al., 2018a; Yang et al., 2019; Tu et al., 2019; Lin et al., 2019), as they are able to effectively collect relevant evidence based on an information aggregation scheme. Specifically, our model is based on graph convolutional networks (GCNs) (Kipf and Welling, 2017) to perform reasoning.
Formally, given a graph  $\mathbf{G}$  with  $n$  nodes, which can be represented with an  $n\times n$  adjacency matrix  $\mathbf{A}$  induced by the previous structure induction module, the convolution computation for the node  $i$  at the  $l$ -th layer, which takes the representation  $\mathbf{u}_i^{l - 1}$  from previous layer as input and outputs the updated representations  $\mathbf{u}_i^l$ , can be defined as:
$$
\mathbf {u} _ {i} ^ {l} = \sigma \left(\sum_ {j = 1} ^ {n} \mathbf {A} _ {i j} \mathbf {W} ^ {l} \mathbf {u} _ {i} ^ {l - 1} + \mathbf {b} ^ {l}\right) \tag {9}
$$
where  $\mathbf{W}^l$  and  $\mathbf{b}^l$  are the weight matrix and bias vector for the  $l$ -th layer, respectively.  $\sigma$  is the ReLU (Nair and Hinton, 2010) activation function.  $\mathbf{u}_i^0 \in \mathbb{R}^d$  is the initial contextual representation of the  $i$ -th node constructed by the node constructor.
Following Guo et al. (2019b), we use dense connections to the GCNs in order to capture more structural information on a large document-level graph. With the help of dense connections, we are able to train a deeper model, allowing richer local and non-local information to be captured for learning a better graph representation. The computations on each graph convolution layer is similar to Equation (9).
Though structured attention (Kim et al., 2017; Liu and Lapata, 2018) is able to automatically induce a latent structure, recent research efforts show that the induced structure is relatively shallow and may not be able to model the complex dependencies for document-level input (Liu et al., 2019b; Ferracane et al., 2019). Unlike previous work (Liu and Lapata, 2018) that only induces the latent structure once, we repeatedly refine the document-level graph based on the updated representations, allowing the model to infer a more informative structure that goes beyond simple parent-child relations.
As shown in Figure 3, we stack  $N$  blocks of the dynamic reasoner in order to induce the document-level structure  $N$  times. Intuitively, the reasoner
induces a shallow structure at early iterations since the information propagates mostly between neighboring nodes. As the structure gets more refined by interactions with richer non-local information, the induction module is able to generate a more informative structure.
2.3 Classifier
After  $N$  times of refinement, we obtain representations of all the nodes. Following Yao et al. (2019), for each entity pair  $(\mathbf{e}_i,\mathbf{e}_j)$  , we use a bilinear function to compute the probability for each relation type  $r$  as:
$$
P (r \mid \mathbf {e} _ {i}, \mathbf {e} _ {j}) = \sigma \left(\mathbf {e} _ {i} ^ {T} \mathbf {W} _ {\mathbf {e}} \mathbf {e} _ {j} + \mathbf {b} _ {e}\right) _ {r} \tag {10}
$$
where  $\mathbf{W}_e\in \mathbb{R}^{d\times k\times d}$  and  $\mathbf{b}_e\in \mathbb{R}^k$  are trainable weights and bias, with  $k$  being the number of relation categories,  $\sigma$  is the sigmoid function, and the subscript  $r$  in the right side of the equation refers to the relation type.
We evaluate our model on DocRED (Yao et al., 2019), the largest human-annotated dataset for document-level relation extraction, and another two popular document-level relation extraction datasets in the biomedical domain, including Chemical-Disease Reactions (CDR) (Li et al., 2016a) and Gene-Disease Associations (GDA) (Wu et al., 2019). DocRED contains 3,053 documents for training, 1,000 for development and 1,000 for test, totally with 132,375 entities and 56,354 relational facts. CDR consists of 500 training instances, 500 development instances, and 500 testing instances. GDA contains 29,192 documents for training and 1,000 for test. We follow (Christopoulou et al., 2019) to split training set of GDA into an 80/20 split for training and development.
With more than  $40\%$  of the relational facts requiring reading and reasoning over multiple sentences, DocRED significantly differs from previous sentence-level datasets (Doddington et al., 2004; Hendrickx et al., 2009; Zhang et al., 2018). Unlike existing document-level datasets (Li et al., 2016a; Quirk and Poon, 2017; Peng et al., 2017; Verga et al., 2018; Jia et al., 2019) that are in the specific biomedical domain considering only the drug-genel-disease relation, DocRED covers a broad range of categories with 96 relation types.
Batch size	20
Learning rate	0.001
Optimizer	Adam
Hidden size	120
Induction block number	2
GCN dropout	0.3
Table 1: Hyper-parameters of LSR.
3.2 Setup
We use  $\mathsf{spaCy}^2$  to get the meta dependency paths of sentences in a document. Following Yao et al. (2019) and Wang et al. (2019), we use the GloVe (Pennington et al., 2014) embedding with BiLSTM, and Uncased BERT-Base (Devlin et al., 2019) as the context encoder. All hyper-parameters are tuned based on the development set. We list some of the important hyper-parameters in Table 1.
Following Yao et al. (2019), we use  $F_{1}$  and Ign  $F_{1}$  as the evaluation metrics. Ign  $F_{1}$  denotes  $F_{1}$  scores excluding relational facts shared by the training and dev/test sets.  $F_{1}$  scores for intra- and inter-sentence entity pairs are also reported. Evaluation on the test set is done through CodaLab<sup>3</sup>.
We compare our proposed LSR with the following three types of competitive models on the DocRED dataset, and show the main results in Table 2.
- Sequence-based Models. These models leverage different neural architectures to encode sentences in the document, including convolutional neural networks (CNN) (Zeng et al., 2014), LSTM, bidirectional LSTM (BiLSTM) (Cai et al., 2016) and attention-based LSTM (ContextAware) (Sorokin and Gurevych, 2017).
- Graph-based Models. These models construct task-specific graphs for inference. GCNN (Sahu et al., 2019) constructs a document-level graph by co-reference links, and then applies relational GCNs for reasoning. EoG (Christopoulou et al., 2019) is the state-of-the-art document-level relation extraction model in biomedical domain. EoG first uses heuristics to construct the graph, then leverages an edge-oriented model to perform inference. GCNN and EoG are based on static structures. GAT (Veličković et al., 2018) is able to learn the weighted graph structure based on a local attention mechanism. AGGCN (Guo
$^{2}$ https://spacy.io/
3https://competitions.codalab.org/competitions/20717
Model	Dev	Test
Ign F1	F1	Intra-F1	Inter-F1	Ign F1	F1
CNN (Yao et al., 2019)	41.58	43.45	51.87*	37.58*	40.33	42.26
LSTM (Yao et al., 2019)	48.44	50.68	56.57*	41.47*	47.71	50.07
BiLSTM (Yao et al., 2019)	48.87	50.94	57.05*	43.49*	48.78	51.06
ContextAware (Yao et al., 2019)	48.94	51.09	56.74*	42.26*	48.40	50.70
GCNN♣ (Sahu et al., 2019)	46.22	51.52	57.78	44.11	49.59	51.62
EoG♣ (Christopoulou et al., 2019)	45.94	52.15	58.90	44.60	49.48	51.82
GAT♣ (Veličković et al., 2018)	45.17	51.44	58.14	43.94	47.36	49.51
AGGCN♣ (Guo et al., 2019a)	46.29	52.47	58.76	45.45	48.89	51.45
GloVe+LSR	48.82	55.17	60.83	48.35	52.15	54.18
BERT (Wang et al., 2019)	-	54.16	61.61*	47.15*	-	53.20
Two-Phase BERT (Wang et al., 2019)	-	54.42	61.80*	47.28*	-	53.92
BERT+LSR	52.43	59.00	65.26	52.05	56.97	59.05
Table 2: Main results on the development and the test set of DocRED: Models with  $\clubsuit$  are adapted to DocRED based on their open implementations. Results with  $*$  are computed based on re-trained models as we need to evaluate  $F_{1}$  for both intra- and inter-sentence setting, which are not given in original papers.
et al., 2019a) is the state-of-the-art sentence-level relation extraction model, which constructs the latent structure by self-attention. These two models are able to dynamically construct task-specific structures.
- BERT-based Models. These models fine-tune BERT (Devlin et al., 2019) for DocRED. Specifically, Two-Phase BERT (Wang et al., 2019) is the best reported model. It is a pipeline model, which predicts if the relation exists between entity pairs in the first phase and predicts the type of the relation in the second phase.
As shown in Table 2, LSR with GloVe achieves  $54.18F_{1}$  on the test set, which is the new state-of-the-art result for models with GloVe. In particular, our model consistently outperforms sequence-based models by a significant margin. For example, LSR improves upon the best sequence-based model BiLSTM by 3.1 points in terms of  $F_{1}$ . This suggests that models which directly encode the entire document are unable to capture the inter-sentence relations present in documents.