SlowGuess/NAACL_2022_OCR huggingface dataset:
Yiwei Wang<sup>1</sup>
Muhao Chen
Yuxuan Liang
Wenxuan Zhou
Yujun Cai3
Bryan Hooi
<sup>2</sup> University of Southern California
<sup>3</sup>Nanyang Technological University
wangyw_seu@foxmail.com
Entity types and textual context are essential properties for sentence-level relation extraction (RE). Existing work only encodes these properties within individual instances, which limits the performance of RE given the insufficient features in a single sentence. In contrast, we model these properties from the whole dataset and use the dataset-level information to enrich the semantics of every instance. We propose the GRAPHCACHE (Graph Neural Network as Caching) module, that propagates the features across sentences to learn better representations for RE. GRAPHCACHE aggregates the features from sentences in the whole dataset to learn global representations of properties, and use them to augment the local features within individual sentences. The global property features act as dataset-level prior knowledge for RE, and a complement to the sentence-level features. Inspired by the classical caching technique in computer systems, we develop GRAPHCACHE to update the property representations in an online manner. Overall, GRAPHCACHE yields significant effectiveness gains on RE and enables efficient message passing across all sentences in the dataset.
Sentence-level relation extraction (RE) aims at identifying the relationship between two entities mentioned in a sentence. RE is crucial to the structural perception of human language, and also benefits many NLP applications such as automated knowledge base construction (Distiawan et al., 2019), event understanding (Wang et al., 2020a), discourse understanding (Yu et al., 2020), and question answering (Zhao et al., 2020). The modern tools of choice for RE are the large-scale pretrained language models (PLMs) that are used to encode individual sentences, therefore obtaining the sentence-level representations (Liu et al., 2019; Joshi et al., 2020; Yamada et al., 2020).
Existing work considers entity types and textual context as essential properties for RE (Peng et al., 2020; Peters et al., 2019; Zhou and Chen, 2021). Nonetheless, most existing RE models only capture these properties locally within individual instances, while not globally modeling them from the whole dataset. Given the insufficient features of a single sentence, it is beneficial to model these properties from the whole dataset and use them to enrich the semantics of individual instances.
To overcome the aforementioned limitation, we propose to mine the entity and contextual information beyond individual instances so as to further improve the relation representations. Particularly, we first construct a heterogeneous graph to connect the instances sharing common properties for RE. This graph includes the sentences and property caches. Each cache represents a property of entity types or contextual topics. We connect every sentence to the corresponding property caches (see Fig. 1), and perform message passing over edges based on a graph neural network (GNN). In this way, the property caches aggregate the features from connected sentences, which will act as a complement to the sentence-level features and provide prior knowledge when identifying relations.
The constructed graph connecting sentences has the same scale as the whole dataset, which leads to high computational complexity of the GNN. To address this issue, our idea is to view the message passing of GNNs as data loading in computer systems, adapting the classical caching techniques to efficiently mining the property information from all sentences. We encapsulate this computational idea in a new GNN module, called GRAPHCACHE (Graph Neural Network as Caching), that uses an online updating strategy to refresh the property caches' representations. In addition, we design an attention-based global-local fusion module to augment the sentence-level representations using the property caches with adaptive weights.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1698 - 1708
Figure 1: We construct a heterogeneous graph to connect the sentences sharing common properties for RE. We consider two kinds of properties: contextual topics and entity types.
GRAPHCACHE can be incorporated into popular RE models to improve their effectiveness without increasing their time complexity, as analyzed in theory (§3.2). As far as we know, ours is the first work to propagate the features across instances to enrich the semantics for sentence-level RE. We evaluate GRAPHCACHE on three public RE benchmarks including TACRED (Zhang et al., 2017), SemEval-2010 task 8 (Hendrickx et al., 2019), and TACREV (Alt et al., 2020a). Empirical results show that GRAPHCACHE consistently improves the effectiveness of popular RE models by a significant margin and propagates features between all sentences in an efficient manner.
Sentence-Level Relation Extraction. Early research efforts (Zeng et al., 2014; Wang et al., 2016; Zhang et al., 2017) train RE models from scratch based on lexicon-level features. Recent work has shifted to fine-tuning pretrained language models (PLMs; Devlin et al. 2019; Liu et al. 2019) resulting in better performance. For example, BERT-MTB (Baldini Soares et al., 2019) continually finetunes the PLM with a matching-the-blanks objective that decides whether two sentences share the same entity. SpanBERT (Joshi et al., 2020) pretrains a masked language model on random contiguous spans to learn span-boundaries and predict the entire masked span. LUKE (Yamada et al., 2020) extends the PLM's vocabulary with entities from Wikipedia and proposes an entity-aware self-attention mechanism. K-Adapter (Wang et al., 2020b) fixes the parameters of the PLM and uses feature adapters to infuse factual and linguistic knowledge. Despite their effectiveness, most exist
ing work on sentence-level RE exploits the entity information and context within only an individual instance, while we propose to globally capture the semantic information from the whole dataset to augment the relation representations. Our model can be flexibly plugged into existing RE models and improve their effectiveness without increasing the time complexity.
Graph Neural Networks for Natural Language Processing. Due to the large body of work on applying GNNs to NLP, we refer readers to a recent survey (Wu et al., 2021) for a general review. GNNs have been explored in several NLP tasks such as semantic role labeling (Marcheggiani and Titov, 2017), machine translation (Bastings et al., 2017), and text classification (Henaff et al., 2015; Defferrard et al., 2016; Kipf and Welling, 2016; Peng et al., 2018; Yao et al., 2019). GNNs have also been widely adopted in various variants of relation extraction on the sentence level, (Zhang et al., 2018; Zhu et al., 2019; Guo et al., 2019a), the document level (Sahu et al., 2019; Christopoulou et al., 2019; Nan et al., 2020; Zeng et al., 2020), and the dialogue level (Xue et al., 2021). However, on the sentence-level relation extraction, most existing work (Zhang et al., 2018; Guo et al., 2019b; Wu et al., 2019) uses the graph neural networks to encode the relation representations from individual instances instead of operating the message passing between instances. In contrast, we build a heterogeneous graph to connect the instances that share the properties for RE, and design the caching updater to efficiently perform the message passing between instances.
Task Definition. Sentence-level relation extraction (RE) aims to identify the relation between a pair of entities in a sentence. In this task, each instance is composed of a sentence, the subject and object entities, and entity types. For example, in the sentence 'Mary gave birth to Jerry at the age of 21.' $^{1}$ , 'Mary' and 'Jerry' are the entities, the entity types are both person, and the ground-truth relation between 'Jerry' and 'Mary' is parent.
We propose GRAPHCACHE (Graph Neural Networks as Caching) as a message passing methodology to model the dataset-level property repre
<sup>1</sup>We use underline and wavy line to denote subject and object respectively by default.
sentations and use them to enrich every instance's semantics. GRAPHCACHE creates a graph representation where sentences with shared property information are connected with property caches. GRAPHCACHE first models the global semantic information by aggregating the features from the whole dataset, and then fuses the global and local features to augment the relational representations for every sentence.
We analogize the message passing in GNNs to caching in computer systems. Caching is about loading data from high volume disks to low volume caches, so as to accelerate data loading. Analogously, when GNNs perform the message passing between sentences through a smaller number of bridge nodes, we can think of the massive sentences in the dataset as the disk data, and the properties, which aggregates the features from sentences, as caches. GRAPHCACHE can be flexibly plugged into existing RE models. As far as we know, ours is the first work to propagate the features between instances to enrich the semantics for RE. GRAPH-CACHE takes an existing RE model as the backbone, e.g., BERT, and takes the sentence-level representations given by the backbone as the inputs of message passing.
A GRAPHCache module consists of three key components: (i) A graph construction technique builds a few property caches. Each cache represents a property for RE: entity type or contextual topic. We connect each sentence to its corresponding properties, so that every property aggregates the features from its neighbor sentences. (ii) Caching message passing aggregates the sentence-level representations to model the properties' representations in an online manner. (iii) Global-local fusion fuses the global property representations and local sentence-level ones to augment the relation representations. Next, we will discuss the three main components in more detail.
3.1 Graph Construction for Sentence-level Relation Extraction
We build a large and heterogeneous graph to connect the sentences sharing the properties: entity types and textual context, which are essential for RE (Peng et al., 2020; Peters et al., 2019; Zhou and Chen, 2021). The heterogeneous graph is defined as  $G = (\mathcal{V}, \mathcal{E})$ , where  $\mathcal{V}$  is the set of nodes, and  $\mathcal{E}$  is the set of edges.  $\mathcal{V} = \mathcal{V}_S \cup \mathcal{V}_P$ , where  $\mathcal{V}_S$  is the set of sentences, and  $\mathcal{V}_P = \mathcal{V}_C \cup \mathcal{V}_E$  is the property
caches. Here  $\mathcal{V}_C$  is the set of latent topics (Zeng et al., 2018) mined from the latent topics from the text corpus using LDA (Blei et al., 2003), which has been found effective in modeling useful contextual patterns (Jelodar et al., 2019). Each topic is represented by a probability distribution over the words, and we assign each sentence to the top  $P$  topics with the largest probabilities.  $\mathcal{V}_E$  is the set of entity types, where every cache represents the types of an entity pair. The entity types are also crucial for predicting relations (Peng et al., 2020; Zhou and Chen, 2021). An edge  $(p,s)\in \mathcal{E}$  exists if the sentence  $s\in \mathcal{V}_S$  has the property  $p\in \mathcal{V}_P$ .
We will implement a GNN on this graph. Specifically, to incorporate the global property information into relation extraction, the property caches aggregates the features from the connected neighboring sentences. This step enables property caches to globally model the properties from the whole dataset. We then use the global property representations from the caches to enrich every sentence's semantics. In this way, the property caches act as prior knowledge when identifying relations and provide each sentence with more representative features.
3.2 Caching Message Passing
We take an existing RE model as the backbone, e.g., BERT (Devlin et al., 2019), which produces the sentence-level representation as  $\mathbf{h}_s$ . Next, we deploy a two-layer GNN on our heterogeneous graph for message passing across sentences. Specifically, the first GNN layer aggregates the sentence-level representations to property caches at the  $t$ th training step:
$$
\bar {\mathbf {h}} _ {p} (t) = \operatorname {M E A N} \left(\left\{\mathbf {h} _ {s} (t), s \in \mathcal {N} (p) \right\}\right),
$$
$$
\mathbf {h} _ {p} (t) = \mathrm {F F N} \left(\bar {\mathbf {h}} _ {p} (t)\right), \tag {1}
$$
where  $p \in \mathcal{V}_P$  is a property,  $s \in \mathcal{N}(p)$  is a sentence having property  $p$ ,  $\mathrm{MEAN}(\cdot)$  is the mean aggregator (Hamilton et al., 2017), and  $\mathrm{FFN}(\cdot)$  is the feed-forward network.  $\mathrm{FFN}(\cdot)$  can be a linear layer in SGC (Wu et al., 2019), a linear layer followed by a nonlinear activation function in GraphSAGE (Hamilton et al., 2017), or a multi-layer perception in GIN (Xu et al., 2018), etc. We follow SGC (Wu et al., 2019) to implement  $\mathrm{FFN}(\cdot)$  by default. For each property  $p$ , this layer aggregates the sentence-level representations  $\mathbf{h}_s(t)$  from  $s \in \mathcal{N}(p)$  to obtain a global property embedding  $\mathbf{h}_p(t)$ . In this way, the generalized context of each
Figure 2: (left) Existing models encode individual instances for RE. (middle) In standard GNNs (Kipf and Welling, 2016), we predict for an instance by aggregating the features from many other sentences in the dataset, leading to high time complexity. (right) Our GRAPHCache implements a caching updater (see Eq. 2) to update the properties' representations in an online manner, which significantly reduces the time complexity.
Algorithm 1 GRAPHCACHE for Relation Extraction
Input: The number of training steps  $T$ , the dataset  $\mathcal{D} = \{se_s, r_s | s = 1, 2, \dots, N\}$ , where  $se_s, r_s$  are the sentence and relation of the  $s$ th instance, our graph  $G$  defined in §3.1, and the batch size  $B$ .
Output: The model's trained parameters.
1: Initialize the model's parameters as random values, and initialize the values of memory  $\mathcal{M}$  and property caches  $\hat{\mathbf{h}}_p(t)$  as zero.
2: for  $t \gets 1$  to  $T$  do
3: Sample a batch  $\mathcal{B}(t)$  from  $\mathcal{D}$ .
4: for  $s$  in  $\mathcal{B}(t)$  do
5:  $\mathbf{h}_s(t) \gets \text{Backbone}(se_s)$
6: end for
7: for  $p$  in  $\mathcal{V}_p$  do
8: Update  $\hat{\mathbf{h}}_p(t)$  as Eq. 2.
9:  $\mathbf{h}_p(t) \gets \mathrm{FFN}(\hat{\mathbf{h}}_p(t))$  as Eq. 1.
10: end for
11: for  $s$  in  $\mathcal{B}(t)$  do
12: Update  $\hat{r}_s(t)$  as Eq. 7.
13:  $\mathcal{M}[s] \gets \mathbf{h}_s(t)$ .
14: end for
15: Back-propagate to update the parameters by minimizing the cross entropy loss between  $\hat{r}_s(t)$  and  $r_i$  of instances in  $\mathcal{B}$ .
16: end for
property is captured from the whole dataset, which is further used to enhance the relation representations for each sentence in the second GNN layer. We describe the details of the second GNN layer in §3.3.
Recall our heterogeneous graph for RE defined in §3.1. At each training step, classical GNNs
perform message passing across edges between the sentences and properties. In this case, the time complexity of the first GNN layer at each training step is  $|\mathcal{E}|$ . Note that  $|\mathcal{E}|$  is larger than  $|V_{s}|$ , which is the number of sentences in the dataset. This leads to poor scalability of GNN, since  $|V_{s}|$  is large in practice.
To address this efficiency issue, we propose Caching GNN for RE in Alg. 1. Our GRAPH-CACHE implements a memory dictionary  $\mathcal{M}$  to store the sentence-level representations from the backbone. To keep consistency with the updating parameters during training, we deploy a caching updater to refresh the properties' representations at each training step:
$$
\begin{array}{l} \hat {\mathbf {h}} _ {p} (t) \\ = \operatorname {U p d a t e r} \left(\hat {\mathbf {h}} _ {p} (t - 1), \left\{\mathbf {h} _ {s} (t), s \in \mathcal {B} (t) \right\}\right) \\ = \hat {\mathbf {h}} _ {p} (t - 1) + \sum_ {s \in \mathcal {N} (p) \cap \mathcal {B} (t)} \frac {\mathbf {h} _ {s} (t) - \mathcal {M} [ s ]}{| \mathcal {N} (p) |}, \tag {2} \\ \end{array}
$$
where  $\mathcal{B}(t)$  denotes the batch at the  $t$ th training step. By doing so, GRAPHCache greatly reduces the time complexity from  $|\mathcal{E}|$  to  $|\mathcal{B}(t)|$  at each training step by using Updater to obtain the property caches' representations  $\hat{\mathbf{h}}_p(t)$ .
Our caching updater is much more efficient than the classical message passing of GNNs, since  $|\mathcal{B}(t)| \ll |V_s| < |\mathcal{E}|$  generally holds in practice. When we aggregate the sentence-level representations from  $\mathcal{M}$ , we provide the following proposition to show that our cache updater is as effective as the first GNN layer in Eq. 1.
Proposition 1. At the  $t$ th training step, denote the property caches' representations in the first GNN layer (see Eq. 1) as  $\bar{\mathbf{h}}_p(t)$ , and those returned by
our updater in Eq. 2 as  $\hat{\mathbf{h}}_p(t)$ . There is  $\hat{\mathbf{h}}_p(t) = \bar{\mathbf{h}}_p(t)$  for  $\forall p\in \mathcal{V}_P,t > 0$
Proof. When  $t > 1$ , if  $\hat{\mathbf{h}}_p(t - 1) = \bar{\mathbf{h}}_p(t - 1)$ , we have:
$$
\begin{array}{l} \hat {\mathbf {h}} _ {p} (t) \\ = \operatorname {U p d a t e r} \left(\hat {\mathbf {h}} _ {p} (t - 1), \left\{\mathbf {h} _ {s} (t), s \in \mathcal {B} (t) \right\}\right) (3) \\ = \hat {\mathbf {h}} _ {p} (t - 1) + \sum_ {s \in \mathcal {N} (p) \cap \mathcal {B} (t)} \frac {\mathbf {h} _ {s} (t) - \mathcal {M} [ s ]}{| \mathcal {N} (p) |} \\ = \sum_ {s \in \mathcal {N} (p)} \frac {\mathcal {M} [ s ]}{| \mathcal {N} (p) |} + \sum_ {s \in \mathcal {N} (p) \cap \mathcal {B} (t)} \frac {\mathbf {h} _ {s} (t) - \mathcal {M} [ s ]}{| \mathcal {N} (p) |} (4) \\ = \sum_ {s \in \mathcal {N} (p) \backslash \mathcal {B} (t)} \frac {\mathcal {M} [ s ]}{| \mathcal {N} (p) |} + \sum_ {s \in \mathcal {N} (p) \cap \mathcal {B} (t)} \frac {\mathbf {h} _ {s} (t)}{| \mathcal {N} (p) |} (5) \\ = \bar {\mathbf {h}} _ {p} (t). (6) \\ \end{array}
$$
Besides, because  $\hat{\mathbf{h}}_p(0) = \bar{\mathbf{h}}_p(0)$  for  $\forall p\in \mathcal{V}_P$  holds as initialized in Alg. Alg. 1, we have  $\hat{\mathbf{h}}_p(t) = \bar{\mathbf{h}}_p(t)$  for  $\forall p\in \mathcal{V}_P,t > 0$
3.3 Global-Local Fusion
In the second GNN layer, we propagate the properties' representations from the property cache to their neighboring sentences in the batch. Since a sentence  $s$  may have more than one latent topic  $|\mathcal{V}_C \cap \mathcal{N}(i)| > 1$ , we utilize the attention mechanism to enable the target sentence to attend to different topics with adaptive weights.
$$
\mathbf {h} _ {s} ^ {t o p i c} (t) = \mathrm {A t t e n t i o n} (\mathbf {h} _ {s} (t), \{\mathbf {h} _ {p} (t), p \in \mathcal {V} _ {C} \}),
$$
where we follow (Vaswani et al., 2017) to implement Attention. The output  $\mathbf{h}_s^{topic}(t)$  is the topic embedding fused for sentence  $s$ . In this way, a sentence can be trained to attend to more relevant topics with higher weights.
Next, we have the entity type embedding of sentence  $s$  as  $\mathbf{h}_s^{entity}(t) = \mathbf{h}_p(t), p \in \mathcal{V}_E \cap \mathcal{N}(s)$ , where  $p \in \mathcal{V}_E \cap \mathcal{N}(s)$  is the entity type node connected to sentence  $s$ .  $\mathbf{h}_s^{topic}(t)$  and  $\mathbf{h}_s^{entity}(t)$  are the global representations of the properties related to sentence  $s$ , while  $\mathbf{h}_s$  is the local representation of sentence  $s$ . We fuse the global and local representations to enrich the semantics of sentence  $s$  through a sentence-wise head:
$$
\hat {r} _ {i} (t) = \operatorname {H e a d} \left(\mathbf {h} _ {s} (t) \| \mathbf {h} _ {s} ^ {\text {t o p i c}} (t) \| \mathbf {h} _ {s} ^ {\text {e n t i t y}} (t)\right), \tag {7}
$$
Dataset	#Train	#Dev	#Test	#Classes
TACRED	68,124	22,631	15,509	42
SemEval	6,507	1,493	2,717	19
TACREV	68,124	22,631	15,509	42
where  $\parallel$  denotes concatenation. GRAPHCACHE makes sentence-wise relation predictions  $\hat{r}_i(t)$  using a sentence-wise Head, implemented as a multi-layer perception (MLP), analogous to a PointNet (Qi et al., 2017). Since GRAPHCACHE predicts a relation label for each sentence, it can be trained by standard task-specific classification losses, e.g., cross-entropy (Mannor et al., 2005). During inference, we take  $\hat{r}_i(t)$  after convergence as the output for RE.
In this section, we evaluate the effectiveness of our GRAPHCACHE method when incorporated into various RE models. We compare our methods against a variety of strong baselines on the task of sentence-level RE. We closely follow the experimental setting of the previous work (Zhang et al., 2017; Zhou and Chen, 2021; Zhang et al., 2018) to ensure a fair comparison, as detailed below.
Datasets. We use the standard sentence-level RE datasets: TACRED (Zhang et al., 2017), SemEval2010 Task 8 (Hendrickx et al., 2019), and TACREV (Alt et al., 2020b) for evaluation. TACRED contains over 106k mention pairs drawn from the yearly TAC KBP challenge. SemEval does not provide entity type annotations, for which we only construct the topic caches for message passing. Alt et al. (2020b) relabeled the development and test sets of TACRED to build TACREV. The statistics of these datasets are shown in Tab. 1. We follow (Zhang et al., 2017) to use F1-micro as the evaluation metric.
Compared Methods. We compare GRAPHCACHE with the following state-of-the-art RE models: (1) PA-LSTM (Zhang et al., 2017) extends the bi-directional LSTM by incorporating positional information to the attention mechanism. (2) GCN (Zhang et al., 2018) uses a graph convolutional network to gather relevant contextual information along syntactic dependency paths. (3)
Method	TACRED	SemEval	TACREV
PA-LSTM (Zhang et al., 2017)	65.1	82.1	73.3
GCN (Zhang et al., 2018)	64.0	80.7	71.9
C-GCN (Zhang et al., 2018)	66.4	84.2	74.6
C-SGC (Wu et al., 2019)	67.0	84.8	75.1
SpanBERT (Joshi et al., 2020)	70.8	86.1	78.0
RECENT (Lyu and Chen, 2021)	75.2	85.8	83.0
IREBERT (Zhou and Chen, 2021)	72.9	86.4	81.3
LUKE (Yamada et al., 2020)	72.7	87.8	80.6
LUKE + GRAPHCACHE (ours)	74.8	89.1	81.5
IRERoBERTa (Zhou and Chen, 2021)	74.6	87.5	83.2
IRERoBERTa + GRAPHCACHE (ours)	75.5	88.2	84.2
C-GCN (Zhang et al., 2018) combines GCN and LSTM, leading to improved performance than each method alone. (4) C-SGC (Wu et al., 2019) simplifies GCN by removing the nonlinear layers and achieves higher effectiveness. (5) SpanBERT (Joshi et al., 2020) extends BERT by introducing a new pretraining objective of continuous span prediction. (6) RECENT (Lyu and Chen, 2021) restricts the candidate relations based on the entity types. (7) LUKE (Yamada et al., 2020) pretrains the language model on both large text corpora and knowledge graphs and further proposes an entity-aware self-attention mechanism. (8) IRE (Zhou and Chen, 2021) proposes an improved entity representation technique in data preprocessing, which enables RoBERTa to achieve state-of-the-art performance on RE.
Model Configuration. For the hyper-parameters of the considered baseline methods, e.g., the batch size, the number of hidden units, the optimizer, and the learning rate, we set them as those in the original papers. For LDA used in GRAPHCACHE, we set the number of topics  $K$  as 50, and the number of top relevant topics for every sentence  $P$  as 2. For all experiments, we report the median F-1 scores of five runs of training using different random seeds.
We incorporate the GRAPHCACHE framework with LUKE and  $\mathrm{IRE}_{\mathrm{RoBERTa}}$  and report the results in Tab. 2. Our GRAPHCACHE method improves LUKE by  $2.9\%$  on TACREV,  $1.5\%$  on SemEval, and  $1.1\%$  on TACREV in the F1 score. For  $\mathrm{IRE}_{\mathrm{RoBERTa}}$ , GRAPHCACHE leads to the improvement of  $1.2\%$  on TACRED,  $0.8\%$  on SemEval,  $1.2\%$  on Re-TACRED. As a result, our