SlowGuess/ACL_2020_OCR huggingface dataset:
Xu Han $^{1*}$ , Yi Dai $^{1*}$ , Tianyu Gao $^{1}$ , Yankai Lin $^{2}$ , Zhiyuan Liu $^{1\dagger}$ , Peng Li $^{2}$ , Maosong Sun $^{1}$ , Jie Zhou $^{2}$
<sup>1</sup> State Key Lab on Intelligent Technology and Systems, Institute for Artificial Intelligence,
Department of Computer Science and Technology, Tsinghua University, Beijing, China  
 $^{2}$ Pattern Recognition Center, WeChat AI, Tencent Inc., China  
{hanxu17, daiy17, gty16}@ mails.tsinghua.edu.cn  
{liuzy, sms}@mail.tsinghua.edu.cn  
{yankailin, patrickpli, withtomzhou}@tencent.com
Continual relation learning aims to continually train a model on new data to learn incessantly emerging novel relations while avoiding catastrophically forgetting old relations. Some pioneering work has proved that storing a handful of historical relation examples in episodic memory and replaying them in subsequent training is an effective solution for such a challenging problem. However, these memory-based methods usually suffer from overfitting the few memorized examples of old relations, which may gradually cause inevitable confusion among existing relations. Inspired by the mechanism in human long-term memory formation, we introduce episodic memory activation and reconsolidation (EMAR) to continual relation learning. Every time neural models are activated to learn both new and memorized data, EMAR utilizes relation prototypes for memory reconsolidation exercise to keep a stable understanding of old relations. The experimental results show that EMAR could get rid of catastrophically forgetting old relations and outperform the state-of-the-art continual learning models. The code and datasets are released on https://github.com/thunlp/ContinualRE.
Relation extraction aims at detecting relations between entities from text, e.g., extracting the relation "the president of" from the given sentence "Newton served as the president of the Royal Society", which could serve as external resource for various downstream applications (Dong et al., 2015; Xiong et al., 2017; Schlichtkrull et al.,
2018). The conventional RE methods (Riedel et al., 2013; Zeng et al., 2014; Lin et al., 2016) mostly focus on recognizing relations for a fixed pre-defined relation set, and cannot handle rapidly emerging novel relations in the real world.
Some researchers therefore explore to detect and learn incessantly emerging relations in an open scenario. As shown in Figure 1, their efforts can be formulated into a two-step pipeline: (1) Open Relation Learning extracts phrases and arguments to construct patterns of specific relations, and then discovers unseen relation types by clustering patterns, and finally expands sufficient examples of new relation types from large-scale textual corpora; (2) Continual Relation Learning continually uses those expanded examples of new relations to train an effective classifier. The classifier is trained on a sequence of tasks for handling both existing and novel relations, where each task has its own relation set. Although continual relation learning is vital for learning emerging relations, there are rare explorations for this field.
A straightforward solution is to store all historical data and re-train models every time new relations and examples come in. Nevertheless, it is computationally expensive since relations are in sustainable growth. Moreover, the huge example number of each relation makes frequently mixing new and old examples become infeasible in the real world. Therefore, storing all data is not practical in continual relation learning. In view of this, the recent preliminary work (Wang et al., 2019) indicates that the main challenge of continual relation learning is the catastrophic forgetting problem, i.e., it is hard to learn new relations and meanwhile avoid forgetting old relations, considering memorizing all the data is almost impossible.
6429
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6429-6440 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 1: The whole pipeline to detect and learn new relations in an open scenario.
Recent work (Shin et al., 2017; Kemker and Kanan, 2018; Chaudhry et al., 2019) has shown that the memory-based approaches, maintaining episodic memory to save a few training examples in old tasks and re-training memorized examples during training new tasks, are one of the most effective solutions to the catastrophic forgetting problem, especially for continual learning in NLP scenarios (Wang et al., 2019; d'Autume et al., 2019). However, existing memory-based models still suffer from an overfitting problem: when adapting them for continual relation learning, they may frequently change feature distribution of old relations, gradually overfit a few examples in memory, and finally become confused among old relations after long-term training.
In fact, these memory-based methods are similar to long-term memory model of mammalian memory in neuroscience (McClelland et al., 1995; Bontempi et al., 1999). Although researchers in neuroscience are not clear about secrets inside the human brain, they reach a consensus that the formation of long-term memory relies on continually replaying and consolidating information (Tononi and Cirelli, 2006; Boyce et al., 2016; Yang et al., 2014), corresponding to the episodic memory and memory replay in continual learning models. Yet later work (Nader et al., 2000; Lee et al., 2004; Alberini, 2005) in neuroscience indicates that reactivation of consolidated memory triggers a reconsolidation stage to continually maintain memory, and memory is easy to be changed or erased in this stage. To apply some reconsolidation exercises can help memory go through this stage and keep long-term memory stable. Intuitively, the ex
isting memory-based models seem like continual memory activation without reconsolidation exercises, and thus become sensitive and volatile.
Inspired by the reconsolidation mechanism in human long-term memory formation, we introduce episodic memory activation and reconsolidation (EMAR) to continual relation learning in this paper. More specifically, when training models on new relations and their examples, we first adopt memory replay to activate neural models on examples of both new relations and memory, and then utilize a special reconsolidation module to let models avoid excessively changing and erasing feature distribution of old relations. As the core of relation learning is to grasp relation prototypes rather than rote memorization of relation examples, our reconsolidation module requires models to be able to distinguish old relation prototypes after each time memory is replayed and activated. As compared with pioneering explorations to improve episodic memory replay (Chaudhry et al., 2019; Wang et al., 2019), with toughly keeping feature distribution of old relations invariant, EMAR is more flexible in feature spaces and powerful in remembering relation prototypes.
We conduct sufficient experiments on several RE datasets, and the results show that EMAR effectively alleviates the catastrophic forgetting problem and significantly outperforms the state-of-the-art continual learning models. Further experiments and analyses indicate the reasons for the effectiveness of EMAR, proving that it can utilize a few examples in old tasks to reconsolidate old relation prototypes and keep better distinction among old relations after long-term training.
The conventional RE work, including both supervised RE models (Zelenko et al., 2003; Zhou et al., 2005; Gormley et al., 2015; Socher et al., 2012; Liu et al., 2013; Zeng et al., 2014; Nguyen and Grishman, 2015; dos Santos et al., 2015; Xu et al., 2015; Liu et al., 2015; Miwa and Bansal, 2016) and distantly supervised models (Bunescu and Mooney, 2007; Mintz et al., 2009; Riedel et al., 2010; Hoffmann et al., 2011; Zeng et al., 2015; Lin et al., 2016; Han et al., 2018a; Baldini Soares et al., 2019), focuses on extracting predefined relations from text. Yet in the real world, new relations are rapidly emerging, and it is impossible to train models with a fixed dataset once to cover all relations. Hence, some researchers pay their attention to relation learning in various open scenarios, in order to detect and learn relations without pre-defined relation sets. As we introduced before, learning incessantly emerging relations consists of two important steps: open relation learning and continual relation learning.
There have been many efforts for open relation learning, including pattern extraction (Banko et al., 2007; Fader et al., 2011; Mausam et al., 2012; Del Corro and Gemulla, 2013; Angeli et al., 2015; Petroni et al., 2015; Stanovsky and Dagan, 2016; Mausam, 2016; Cui et al., 2018), relation discovery (Yao et al., 2011; Marcheggiani and Titov, 2016), relation clustering (Shinyama and Sekine, 2006; Elsahar et al., 2017; Wu et al., 2019), and data collection (Riloff et al., 1999; Etzioni et al., 2005; Pantel and Pennacchiotti, 2006; Rozenfeld and Feldman, 2008; Nakashole et al., 2011; Zhu et al., 2009; Gao et al., 2020). However, for continual relation learning, there are still only some preliminary explorations for it. Following continual learning setting<sup>1</sup> (Ring, 1994; Thrun and Pratt, 2012) in machine learning, Wang et al. (2019) first explore continual relation learning.
Existing continual learning methods focus on three research directions: (1) consolidation-based methods (Kirkpatrick et al., 2017; Zenke et al., 2017; Li and Hoiem, 2017; Liu et al., 2018; Ritter et al., 2018) which consolidate the model parameters important to previous tasks and reduce their learning weights; (2) dynamic architecture methods (Chen et al., 2016; Rusu et al., 2016; Fernando et al., 2017) which dynamically expand model architectures to learn new tasks and ef
fectively prevent forgetting old tasks. Yet model size growing dramatically with increasing tasks makes these methods unsuitable for NLP applications; (3) memory-based methods (Lopez-Paz and Ranzato, 2017; Rebuffi et al., 2017; Shin et al., 2017; Kemker and Kanan, 2018; Aljundi et al., 2018; Chaudhry et al., 2019) remember a few examples in old tasks and continually learn them with emerging new tasks to alleviate catastrophic forgetting. Among these methods, the memory-based methods have been proven to be the most promising for NLP tasks, including both relation learning (Wang et al., 2019) and other NLP tasks (d'Autume et al., 2019; Sun et al., 2019).
Inspired by reconsolidation in human memory formation, we introduce episodic memory activation and reconsolidation (EMAR) to alleviate the overfitting problem of the existing memory-based methods and better learn relations continually.
3.1 Task Definition and Overall Framework
Continual relation learning trains models on a sequence of tasks, where the  $k$ -th task has its own training set  $\mathcal{T}_k$ , validation set  $\nu_{k}$ , and query set  $\mathcal{Q}_k$ . Each set of the  $k$ -th task, e.g.  $\mathcal{T}_k = \{(x_1^{\mathcal{T}_k},y_1^{\mathcal{T}_k}),\ldots ,(x_N^{\mathcal{T}_k},y_N^{\mathcal{T}_k})\}$ , consists of a series of examples and their corresponding relation labels, where  $N$  is the example number of  $\mathcal{T}_k$ . Each example  $x_{i}^{\mathcal{T}_{k}}$  and its label  $y_{i}^{\mathcal{T}_{k}}$  indicate that  $x_{i}^{\mathcal{T}_{k}}$  can express the relation  $y_{i}^{\mathcal{T}_{k}}\in \mathcal{R}_{k}$ , where  $\mathcal{R}_k$  is the relation set of the  $k$ -th task.
More specifically, models will be trained on  $\mathcal{T}_k$  at the  $k$ -th step to learn the new relations in  $\mathcal{R}_k$ . As relations are emerging and accumulating, continual relation learning requires models to perform well on both the  $k$ -th task and previous  $k-1$  tasks. Hence, after training on  $\mathcal{T}_k$ , models will be evaluated on  $\tilde{\mathcal{Q}}_k = \bigcup_{i=1}^k \mathcal{Q}_i$ , and required to classify each query example into the all known relation set  $\tilde{\mathcal{R}}_k = \bigcup_{i=1}^k \mathcal{R}_i$ . Therefore, the evaluation will be more and more difficult with the growth of tasks.
For handling the catastrophic forgetting in continual relation learning, an episodic memory module  $\mathcal{M} = \{\mathcal{M}_1,\mathcal{M}_2,\ldots \}$  is set to store a few examples of historical tasks, each memory module  $\mathcal{M}_k = \{(x_1^{\mathcal{M}_k},y_1^{\mathcal{M}_k}),\dots ,(x_B^{\mathcal{M}_k},y_B^{\mathcal{M}_k})\}$  stores several examples and labels that come from  $\mathcal{T}_k$  where  $(x_{i}^{\mathcal{M}_{k}},y_{i}^{\mathcal{M}_{k}})\in \mathcal{T}_{k}$  and  $B$  is the constrained memory size for each task.
As shown in Figure 2, when models are trained
<sup>1</sup> Some work names it lifelong or incremental learning.
6431
Figure 2: A simple example of continually learning four tasks (each task has only one relation: A, B, C, D respectively) to demonstrate the overall framework of episodic memory activation and reconsolidation during continual relation learning. The purple solid lines and dotted lines represent the forward and backward propagation respectively. The black dotted lines represent the data flow.
on the  $k$ -th task, our framework includes several steps to learn new relations and meanwhile avoid forgetting old relations: (1) First (Section 3.3), we fine-tune the example encoder on the training set  $\mathcal{T}_k$  of the  $k$ -th task to let the model be aware of new relation patterns. (2) Second (Section 3.4), for each relation in the  $k$ -th relation set  $\mathcal{R}_k$ , we select its informative examples and store the examples into the episodic memory  $\mathcal{M}_k$ . (3) Finally (Section 3.5), we iteratively adopt memory replay and activation as well as memory reconsolidation to learn new relation prototypes while strengthening distinguishing old relation prototypes.
Besides, we will introduce how to train models as well as predict relations for query examples in Section 3.6. As the example encoder is used in all other steps, we first introduce it in Section 3.2 before other steps.
3.2 Example Encoder
Given an example  $x$ , we adopt an example encoder to encode its semantic features for detecting and learning relations. To be specific, we first tokenize the given example into several tokens, and then input the tokenized tokens into neural networks to compute its corresponding embedding. As extracting relations from sentences is related to those entities mentioned in sentences, we thus add special tokens into the tokenized tokens to indicate the beginning and ending positions of those entities. For simplicity, we denote such an example encoding operation as the following equation,
$$
\boldsymbol {x} = f (\boldsymbol {x}), \tag {1}
$$
where  $\pmb{x} \in \mathbb{R}^d$  is the semantic embedding of  $x$ , and  $d$  is the embedding dimension. Note that the encoder is not our focus in this paper, we select bidirectional long short-term memory (BiLSTM) (Bengio et al., 1994) as representative encoders to encode examples. In fact, other neural text encoders like convolutional neural networks (Zeng et al., 2014) and pre-trained language models (Devlin et al., 2019) can also be adopted as example encoders.
3.3 Learning for New Tasks
When the  $k$ -th task is arising, the example encoder has not touched any examples of new relations before, and cannot extract the semantic features of them. Hence, we first fine-tune the example encoder on  $\mathcal{T}_k = \{(x_1^{\mathcal{T}_k},y_1^{\mathcal{T}_k}),\ldots ,(x_N^{\mathcal{T}_k},y_N^{\mathcal{T}_k})\}$  to grasp new relation patterns in  $\mathcal{R}_k$ . The loss function of learning the  $k$ -th task is as follows,
$$
\begin{array}{l} \mathcal {L} (\boldsymbol {\theta}) = - \sum_ {i = 1} ^ {N} \sum_ {j = 1} ^ {| \tilde {\mathcal {R}} _ {k} |} \delta_ {y _ {i} \tau_ {k} = r _ {j}} \times \tag {2} \\ \log \frac {\exp (g (f (x _ {i} ^ {\mathcal {T} _ {k}}) , \boldsymbol {r} _ {j}))}{\sum_ {l = 1} ^ {| \tilde {\mathcal {R}} _ {k} |} \exp (g (f (x _ {i} ^ {\mathcal {T} _ {k}}) , \boldsymbol {r} _ {l}))}, \\ \end{array}
$$
where  $r_j$  is the embedding of the  $j$ -th relation  $r_j \in \tilde{\mathcal{R}}_k$  in the all known relation set  $\tilde{\mathcal{R}}_k$ ,  $g(\cdot, \cdot)$  is the function to compute similarities between embeddings (e.g., cosine similarity), and  $\theta$  is the parameters that can be optimized, including the example encoder parameters and relation embeddings. If  $y_i^{\mathcal{T}_k}$  equals  $r_j$ ,  $\delta_{y_i^{\mathcal{T}_k} = r_j} = 1$ , otherwise
6432
$\delta_{y_i^{\mathcal{T}_k = r_j}} = 0$  . For each new relation, we first randomly initialize its embedding and then optimize Eq. (2).
3.4 Selecting Examples for Memory
After several epochs of learning for new tasks with Eq. (2), we store a few examples from  $\mathcal{T}_k$  into the memory  $\mathcal{M}_k$ . More specifically, we select informative and diverse examples from  $\mathcal{T}_k$  to cover new relation patterns as much as possible, which can make the memory effectively approximate the feature distribution of relations.
After encoding all examples of the  $k$ -th task  $\mathcal{T}_k$  into  $\{\pmb{x}_1^{\mathcal{T}_k},\dots,\pmb{x}_N^{\mathcal{T}_k}\}$ , we apply K-Means to cluster these example embeddings, where the number of clusters is the memory size  $B$ . Then, for each cluster, we select the example closest to the cluster centroid and record which relation these selected examples belong to. We denote this selected example set  $\mathcal{C}_k$ . By counting the example number in  $\mathcal{C}_k$  for each relation, we can describe the relation importance in this task: more selected examples of a relation indicates more importance. As the limited memory size, for those more important relations, we select at least  $\left\lfloor \frac{B}{|\mathcal{R}_k|} \right\rfloor$  examples, yet for those less important ones, we select at most  $\left\lceil \frac{B}{|\mathcal{R}_k|} \right\rceil$  examples. If a relation does not have enough examples to fill its allocated memory, this memory will be re-allocated for other relations.
For each relation, we also use K-Means to cluster its own examples, and the number of current clusters is its allocated example number in the memory. For each cluster, we select the example closest to the cluster centroid, and store this example into the memory  $\mathcal{M}_k$ .
3.5 Replay, Activation and Reconsolidation
After fine-tuning the example encoder for  $\mathcal{T}_k$  and selecting informative examples for  $\mathcal{M}_k$ , we iteratively adopt computing prototypes, memory replay and activation, and memory reconsolidation to strengthen identifying new relation patterns and keep distinguishing old relation patterns.
Computing Prototypes
By combining all examples in the episodic memory, we achieve the whole memory set  $\tilde{\mathcal{M}}_k = \bigcup_{i=1}^k \mathcal{M}_i$ . As we aim to grasp relation prototypes rather than rote memorization of relation examples, for each known relation  $r_i \in \tilde{\mathcal{R}}_k$ , we sample a prototype set  $\mathcal{P}_i = \{x_1^{\mathcal{P}_i}, \ldots, x_{|\mathcal{P}_i|}^{\mathcal{P}_i}\}$ , where each
example  $x_{i}^{\mathcal{P}_{i}}$  comes from  $\tilde{\mathcal{M}}_k$  and its label equals  $r_i$ , and compute its prototype embedding,
$$
\boldsymbol {p} _ {i} = \frac {\sum_ {j = 1} ^ {| \mathcal {P} _ {i} |} f \left(x _ {j} ^ {\mathcal {P} _ {i}}\right)}{| \mathcal {P} _ {i} |}, \tag {3}
$$
where  $\pmb{p}_i$  is the relation prototype embedding of  $r_i\in \tilde{\mathcal{R}}_k$
Memory Replay and Activation
In memory replay and activation, the whole memory set  $\tilde{\mathcal{M}}_k$  and the  $k$ -th training set  $\mathcal{T}_k$  will be combined into an activation set  $\mathcal{A}_k = \tilde{\mathcal{M}}_k \cup \mathcal{T}_k = \{(x_1^{\mathcal{A}_k}, y_1^{\mathcal{A}_k}), \ldots, (x_M^{\mathcal{A}_k}, y_M^{\mathcal{A}_k})\}$  to continually activate models to learn new relations and remember old relations, where  $M$  is the total example number of both  $\tilde{\mathcal{M}}_k$  and  $\mathcal{T}_k$ . The loss function is
$$
\begin{array}{l} \mathcal {L} ^ {\mathcal {A}} (\boldsymbol {\theta}) = - \sum_ {i = 1} ^ {M} \sum_ {j = 1} ^ {| \tilde {\mathcal {R}} _ {k} |} \delta_ {y _ {i} ^ {\mathcal {A} _ {k}} = r _ {j}} \times \tag {4} \\ \log \frac {\exp (g (f (x _ {i} ^ {\mathcal {A} _ {k}}) , \boldsymbol {r} _ {j}))}{\sum_ {l = 1} ^ {| \tilde {\mathcal {R}} _ {k} |} \exp (g (f (x _ {i} ^ {\mathcal {A} _ {k}}) , \boldsymbol {r} _ {l}))}. \\ \end{array}
$$
Memory Reconsolidation
As we mentioned before, just conducting memory replay and activation will lead to the overfitting problem, and in the end, models only remember a handful of memorized examples after long-term training. Meanwhile, the core of learning relations is to grasp relation prototypes rather than rote memorization of relation examples. Hence, every time conducting memory replay and activation to grasp both new and old relations, we adopt a memory reconsolidation module to strengthen this process, which seems like conducting reconsolidation exercises to keep long-term memory stable in the human brain.
For each known relation  $r_i \in \tilde{\mathcal{R}}_k$ , we sample its instance set  $\mathcal{I}_i = \{x_1^{\mathcal{I}_i}, \ldots, x_{|\mathcal{I}_i|}^{\mathcal{I}_i}\}$  as is similar to sampling  $\mathcal{P}_i$ , where each example  $x_i^{\mathcal{I}_i} \in \mathcal{I}_i$  also comes from  $\tilde{\mathcal{M}}_k$  and its label equals  $r_i$ . The loss function of the memory reconsolidation is
$$
\mathcal {L} ^ {\mathcal {R}} (\boldsymbol {\theta}) = - \sum_ {i = 1} ^ {| \tilde {\mathcal {R}} _ {k} |} \sum_ {j = 1} ^ {| \mathcal {I} _ {i} |} \tag {5}
$$
$$
\log \frac {\exp (g (f (x _ {j} ^ {\mathcal {I} _ {i}}) , \boldsymbol {p} _ {i}))}{\sum_ {l = 1} ^ {| \bar {\mathcal {R}} _ {k} |} \exp (g (f (x _ {j} ^ {\mathcal {I} _ {i}}) , \boldsymbol {p} _ {l}))},
$$
where  $\pmb{p}_l$  is the relation prototype embedding of  $r_l\in \tilde{\mathcal{R}}_k$  computed by Eq. (3).
6433
3.6 Training and Prediction
For training the  $k$ -th task, we first use  $\mathcal{L}(\theta)$  to optimize parameters for several epochs. Then, we select examples for the memory, and iteratively optimize parameters with  $\mathcal{L}^{\mathcal{A}}(\theta)$  and  $\mathcal{L}^{\mathcal{R}}(\theta)$  until convergence. More details about the training process are shown in Algorithm 1.
After finishing the  $k$ -th task, for each known relation  $r_i \in \tilde{\mathcal{R}}_k$ , we collect all its memorized examples  $\mathcal{E}_i = \{x_1^{\mathcal{E}_i}, \ldots, x_S^{\mathcal{E}_i}\}$  in the whole memory  $\tilde{\mathcal{M}}_k$ , where  $S$  is the example number of  $r_i$  in the memory, and compute final relation prototype for prediction,
$$
\tilde {\boldsymbol {p}} _ {i} = \frac {\boldsymbol {r} _ {i} + \sum_ {j = 1} ^ {S} f \left(x _ {j} ^ {\mathcal {E} _ {i}}\right)}{1 + S}, \tag {6}
$$
where  $r_i$  is the relation embedding of  $r_i$  used in Eq. (2) and Eq. (4). For each query example  $x$  in  $\tilde{\mathcal{Q}}_k$ , we define its score function for the relation  $r_i$ :
$$
s (x, r _ {i}) = g (f (x), \tilde {\boldsymbol {p}} _ {i}), \tag {7}
$$
where  $\tilde{\pmb{p}}_i$  is the final prototype of the relation  $r_i$  computed by Eq. (6). Finally, the prediction  $y$  for the query  $x$  is calculated by:
$$
y = \underset {r _ {i} \in \tilde {\mathcal {R}} _ {k}} {\arg \max } s (x, r _ {i}). \tag {8}
$$
FewRel	SimpleQ	TACRED
W	A	W	A	W	A
Lower Bound	18.9	20.8	63.2	56.9	12.3	9.5
EWC	27.1	30.2	67.2	59.0	14.5	14.5
GEM	49.2	59.8	84.1	79.6	-	-
AGEM	36.1	42.5	77.6	72.2	15.7	16.0
EMR	51.0	62.0	85.2	80.8	28.7	35.6
EA-EMR	56.6	67.3	87.8	82.4	30.5	40.5
EMAR	66.0	77.9	85.2	83.7	44.5	54.4
Upper Bound	81.9	85.8	88.9	84.1	74.3	77.0
Table 1: Accuracy  $(\%)$  of models on three benchmarks. "W" stands for the Whole performance, and "A" stands for the Average performance. The results of FewRel and SimpleQ come from Wang et al. (2019). The result of TACRED comes from our implemented models.
We carry out our experiments on three benchmark datasets: