SlowGuess/ACL_2020_OCR huggingface dataset:
Finally, in the prediction step, following the prior work (Veyseh et al., 2019), we employ the following vector  $V$  as the overall representation vector to predict the relation between  $w_{s}$  and  $w_{o}$  in  $W$ :  $V = [x_{s}, x_{o}, h_{s}, h_{o}, R_{W}]$ . Note that  $V$  involves the information at different abstract levels for  $W$ , i.e., the raw input level with  $x_{s}$  and  $x_{o}$ , the abstract representation level with  $h_{s}$  and  $h_{o}$
2We tried the KL divergence and the mean square error for this, but cosine similarity achieved better performance.
8025
from CEON-LSTM, and the overall sentence vector  $R_W$ . In our model,  $V$  would be fed into a feed-forward neural network with the softmax layer in the end to estimate the probability distribution  $P(\cdot | W, w_s, w_o)$  over the possible relations for  $W$ . The negative log-likelihood function is then obtained to serve as the loss function for the model:  $L_{label} = -\log P(y | W, w_s, w_o)$  ( $y$  is the golden relation label for  $w_s$  and  $w_o$  in  $W$ ). Eventually, the overall loss function of the model in this work is:
$$
\mathcal {L} = \mathcal {L} _ {\text {l a b e l}} + \alpha \mathcal {L} _ {\text {i m p o r t}} + \beta \mathcal {L} _ {\text {p a t h}} \tag {6}
$$
where  $\alpha$  and  $\beta$  are trade-off parameters. The model is trained with shuffled mini-batching.
4.1 Datasets and Hyper-parameters
We evaluate the models in this work using three benchmark datasets, i.e., ACE 2005, SPOUSE, and SciERC. For ACE 2005, similar to the previous work (Nguyen and Grishman, 2016; Fu et al., 2017; Shi et al., 2018; Veyseh et al., 2019), we use the dataset preprocessed and provided by (Yu et al., 2015) for compatible comparison. There are 6 different domains in this dataset, i.e., (bc, bn, cts, nw, un, and wl), covering text from news, conversations and web blogs. Following the prior work, the union of the domains bn and nw (called news) is used as the training data (called the source domain); a half of the documents in bc is reserved for the development data, and the remainder (cts, wl and the other half of bc) serve as the test data (called the target domains). This data separation facilitates the evaluation of the cross-domain generalization of the models due to the domain difference of the training and test data.
The SPOUSE dataset is recently introduced by (Hancock et al., 2018), involving 22,195 sentences for the training data, 2,796 sentences for the validation data, and 2,697 sentences for the test data. Each sentence in this dataset contains two marked person names (i.e., the entity mentions) and the goal is to identify whether the two people mentioned in the sentence are spouses.
Finally, the SciERC dataset (Luan et al., 2018) annotates 500 scientific abstracts for the entity mentions along with the coreferences and relations between them. For RE, this dataset provides 3,219 sentences in the training data, 455 sentences in the validation data and 974 sentences in the test data.
We fine tune the hyper-parameters for the models in this work on the validation data of the ACE 2005 dataset. The best parameters suggested by this process include: 30 dimensions for the position embeddings and entity type embeddings, 200 hidden units for the CEON-LSTM model and all the other hidden vectors in the model (i.e., the hidden vectors in the final feed-forward neural network (with 2 layers) and the intermediate vectors in the weighted sum vector for  $x_{t}^{\prime}$ ), 1.0 for both loss trade-off parameters  $\alpha$  and  $\beta$ , and 0.001 for the initial learning rate with the Adam optimizer. The batch size is set to 50. Finally, we use either the uncontextualized word embeddings word2vec (with 300 dimensions) or the hidden vectors in the last layer of the BERTbase model (with 768 dimensions) (Devlin et al., 2019) to obtain the pre-trained word embeddings for the sentences (Devlin et al., 2019). We find it better to fix BERT in the experiments. Note that besides this section, we provide some additional analysis for the models in the Appendix.
We fist compare the proposed model (called CEON-LSTM) with the baselines on the popular ACE 2005 dataset. In particular, the four following groups of RE models in the prior work on RE with the ACE 2005 dataset is chosen for comparison:
(i) Feature based models: These models hand-design linguistic features for RE, i.e., FCM, Hybrid FCM, LRFCM, and SVM (Yu et al., 2015; Hendrickx et al., 2010).
(ii) Deep sequence-based models: These models employ deep learning architectures based on the sequential order of the words in the sentences for RE, i.e., log-linear, CNN, Bi-GRU, Forward GRU, Backward GRU (Nguyen and Grishman, 2016), and CNN+DANN (Fu et al., 2017).
(iii) Adversarial learning model: This model, called GSN, attempts to learn the domain-independent features for RE (Shi et al., 2018).
(iv) Deep structure-based models: These models use dependency trees either as the input features or the graphs to structure the network architectures in the deep learning models. The state-of-the-art models of this type include: AGGCN (Attention Guided GCN) (Guo et al., 2019), SACNN (Segment-level Attention-based CNN) (Tran et al., 2019) and DRPC (the Dependency Relation Prediction and Control model) (Veyseh et al., 2019). DRPC has the best reported performance on ACE
8026
System	bc	cts	wl	Avg.
FCM (2015)	61.90	52.93	50.36	55.06
Hybrid FCM (2015)	63.48	56.12	55.17	58.25
LRFCM (2015)	59.40	-	-	-
Log-linear (2016)	57.83	53.14	53.06	54.67
CNN (2016)	63.26	55.63	53.91	57.60
Bi-GRU (2016)	63.07	56.47	53.65	57.73
Forward GRU (2016)	61.44	54.93	55.10	57.15
Backward GRU (2016)	60.82	56.03	51.78	56.21
CNN+DANN (2017)	65.16	-	-	-
GSN (2018)	66.38	57.92	56.84	60.38
C-GCN (2018)	65.55	62.98	55.91	61.48
AGGCN (2019)	63.47	59.70	56.50	59.89
SACNN (2019)	65.06	61.71	59.82	62.20
DRPC (2019)	67.30	64.28	60.19	63.92
CEON-LSTM (ours)	68.55	65.42	61.93	65.30
2005. Note that we obtain the performance of these models on the considered datasets using the actual implementation released by the original papers.
Most of the prior RE work on the ACE 2005 dataset uses the uncontextualized word embeddings (i.e., word2vec) for the initial word representation vectors. In order to achieve a fair comparison with the baselines, we first show the performance of the models (i.e., the F1 scores) on the ACE 2005 test datasets when word2vec is employed for the pre-trained word embeddings in Table 1. The first observation from the table is that the deep structured-based models (e.g., C-GCN, DRPC) are generally better than the deep sequence-based models (e.g., CNN, Bi-GRU) and the feature base models with large performance gaps. This demonstrates the benefits of the syntactic structures that can provide useful information to improve the performance for the deep learning models for RE. We will thus focus on these deep structure-based models in the following experiments. Among all the models, we see that the proposed model CEON-LSTM is significantly better than all the baseline models over different test domains/datasets. In particular, CEON-LSTM is  $1.38\%$  and  $3.1\%$  better than DRPC and SACNN (respectively) on the average F1 scores over different test datasets. These performance improvements are significant with  $p < 0.01$  and clearly demonstrate the effectiveness of the proposed CEON-LSTM model for RE.
In order to further compare CEON-LSTM with the baselines, Table 2 presents the performance of the models when the words are represented by the contextualized word embeddings (i.e., BERT). For this case, we also report the performance of the recent BERT-based model (i.e., Entity-Aware BERT (EA-BERT)) in (Wang et al., 2019) for RE
Table 1: F1 scores of the models on the ACE 2005 test datasets using the word2vec word embeddings.
System	bc	cts	wl	Avg.
C-GCN (2018)	67.02	64.4	58.92	63.44
AGGCN (2019)	65.29	63.65	60.35	63.09
SACNN (2019)	68.52	64.21	62.19	64.97
DRPC (2019)	69.41	65.82	61.65	65.62
EA-BERT (2019)	69.25	61.70	58.48	63.14
CEON-LSTM (ours)	71.58	66.92	65.17	67.89
Table 2: F1 scores of the models on the ACE 2005 test datasets using the BERT word embeddings.
System	SPOUSE	SciERC
C-GCN (word2vec) (2018)	73.52	65.30
AGGCN (word2vec) (2019)	73.51	67.91
SACNN (word2vec) (2019)	72.88	67.54
DRPC (word2vec) (2019)	74.66	68.18
CEON-LSTM (word2vec) (ours)	76.43	69.92
C-GCN (BERT) (2018)	75.18	74.11
AGGCN (BERT) (2019)	76.91	75.77
SACNN (BERT) (2019)	77.98	76.42
DRPC (BERT) (2019)	78.93	77.21
CEON-LSTM (BERT) (ours)	81.01	78.24
Table 3: F1 scores of the models on the SPOUSE and Sci-ERC datasets.
on the ACE 2005 dataset. Comparing the models in Table 2 with the counterparts in 1, it is clear that the contextualized word embeddings can significantly improve the deep structure-based models for RE. More importantly, similar to the case with word2vec, we see that the proposed model CEON-LSTM still significantly outperforms all the baselines models with large performance gaps and  $p < 0.01$ , further testifying to the benefits of the CEON-LSTM model in this work.
Finally, in order to demonstrate the generalization of the proposed model over the other datasets, we show the performance of the models on the two other datasets in this work (i.e., SPOUSE and SciERC) using either word2vec or BERT as the word embeddings in Table 3. The results clearly confirm the effectiveness of CEON-LSTM as it is significantly better than all the other models over different datasets and word embedding settings.
The Effect of the Model Components: There are three major components in the proposed model: (1) the introduction of the overall sentence representation  $x_{t}^{\prime}$  into the ON-LSTM cells (called SCG - Sentence Context for Gates), (2) the consistency constraint for the syntax-based and model-based importance scores (called SMC - Syntax-Semantic Consistency), and (3) the similarity constraint for the representation vectors of the overall sentence and the shortest dependency path (called SDPS - Sentence-Dependency Path Similarity). In order to
System	P	R	F1
CEON-LSTM (Full)	74.51	67.29	71.08
- SCG	74.00	66.98	70.45
- SMC	72.87	66.85	69.89
- SDPS	73.02	66.00	69.18
- SCG - SMC	71.52	64.62	68.08
- SCG - SDPS	70.33	64.22	67.17
- SMC - SDPS	71.02	63.95	67.58
- SCG - SMC - SDPS	70.51	63.01	66.98
evaluate the contribution of these components for the overall model CEON-LSTM, we incrementally remove these components from CEON-LSTM and evaluate the performance of the remaining model. Table 4 reports the performance of the models on the ACE 2005 development dataset.
It is clear from the table that all the components are necessary for the proposed model as excluding any of them would hurt the performance significantly. It is also evident that removing more components results in more performance drop, thus demonstrating the complementary nature of the three proposed components in this work.
The Variants for CEON-LSTM: We study several variants of SCG, SMC, and SDPS in CEON-LSTM to demonstrate the effectiveness of the designed mechanisms. In particular, we consider the following alternatives for CEON-LSTM:
(i) Bi-ON-LSTM: Instead of employing the attention-based representation vectors  $x_{t}^{\prime}$  to capture the context of the entire input sentence for the model-based importance scores in SCG, we run two unidirectional ON-LSTM models (i.e., the forward and backward ON-LSTM) to compute the forward and backward importance scores for each word in  $W$ . The final model-based importance score for each word is then the average of the corresponding forward and backward scores.
(ii) SA-ON-LSTM: In this method, instead of using the hidden vector  $h_{t - 1}$  as the query vector to compute the attention weight  $\alpha_{ti}$  in Equation 3 for SCG, we utilize the input representation vector  $x_{t}$  for  $w_{t}$  as the query vector (i.e., replace  $h_{t - 1}$  with  $x_{t}$  in Equation 3). Consequently, SA-ON-LSTM is basically a composed model where we first run the self-attention (SA) model (Vaswani et al., 2017) over  $X$ . The results are then fed into ON-LSTM to obtain the model-based importance scores  $mod_{t}$ .
(iii) CE-LSTM: This aims to explore the effic
Table 4: Ablation study on the development set of ACE 2005. The components listed in each row are removed from the overall model.
System	P	R	F1
CEON-LSTM (proposed)	74.51	67.29	71.08
Bi-ON-LSTM	72.65	67.17	69.28
SA-ON-LSTM	73.21	67.31	70.13
CE-LSTM	71.58	64.19	67.92
EP-ON-LSTM	71.03	65.16	68.45
SP-CEON-LSTM (RW in V)	73.58	66.92	70.13
SP-CEON-LSTM (RW not in V)	72.94	65.21	69.51
Table 5: Models' performance on the development dataset of ACE 2005.
tiveness of ON-LSTM for our model. In CE-LSTM, we replace the ON-LSTM network with the usual LSTM model in CEON-LSTM. The SMC component is not included in this case as the LSTM model cannot infer the importance scores.
(iv) EP-ON-LSTM: Before this work, the DRPC model in (Veyseh et al., 2019) has the state-of-the-art on ACE 2005. Both DRPC and CEON-LSTM apply a more general representation of the dependency trees in a deep learning model (i.e., avoid directly using the original trees to improve the generalization). To illustrate the benefit of the importance score representation for SMC, EP-ON-LSTM replaces the importance score representation for the dependency trees in CEON-LSTM with the dependency edge representation in DRPC. In particular, we replace the term  $L_{import}$  in the overall loss function (i.e., Equation 6) with the dependency edge prediction loss (using the ON-LSTM hidden vectors) in DRPC for EP-ON-LSTM.
(v) SP-CEON-LSTM: This model removes the SDPS component and includes the representation vector of the dependency path  $DP$  (i.e.,  $R_{DP}$ ) in the final representation  $V$  for relation prediction. We consider both retaining and excluding the sentence representation  $R_W$  in  $V$  in this case. This model seeks to show that the use of  $R_{DP}$  for the similarity encouragement with  $R_W$  is more effective than employing  $R_{DP}$  directly in  $V$ .
Table 5 reports the performance of these CEON-LSTM variations on the ACE 2005 development dataset. As we can see from the table, all the considered variants have significantly worse performance than CEON-LSTM (with  $p < 0.005$ ). This clearly helps to justify the designs of the components SCG, SMC and SDPS for CEON-LSTM in this work.
Baseline for the Model-Based Importance Scores: One of the contributions in our work is to employ the gates in the cells of ON-LSTM to obtain the model-based importance scores that are then used to promote the consistency with the syntax-based importance scores (i.e., in the SMC compo
System	P	R	F1
CEON-LSTM (proposed)	74.51	67.29	71.08
HIS-CEON-LSTM	72.02	63.97	68.29
Table 6: Models' performance on the development dataset of ACE 2005.
nent). In order to demonstrate the effectiveness of the master cell gates to obtain the model-based importance scores, we evaluate a typical baseline where the model-based importance score modi for  $w_{i}\in W$  is computed directly from the hidden vector  $h_i$  of CEON-LSTM (i.e., by feeding  $h_i$  into a feed-forward neural network with sigmoid activation function in the end). The model-based importance scores obtained in this way then replace the importance scores from the cell gates and are used in the SMC component of CEON-LSTM in the usual way (i.e., via the KL divergence in  $L_{import}$ ) (note that we tried the alternatives for the KL divergence in  $L_{import}$  (i.e., the mean square error and the cosine similarity between the syntax-based and model-based importance scores), but the KL divergence produced the best results for both CEON-LSTM and HIS-CEON-LSTM on the development data). The resulting model is called HIS-CEON-LSTM. Table 6 reports the performance of HIS-CEON-LSTM and the proposed model CEON-LSTM on the ACE 2005 development dataset. It is clear from this table that the proposed model CEON-LSTM achieves significantly better performance than HIS-CEON-LSTM (with large performance gap), thus testifying to the importance of the master gates to obtain the model-based importance scores for CEON-LSTM.
We introduce a new deep learning model for RE (i.e., CEON-LSTM) that features three major proposals. First, we represent the dependency trees via the syntax-based importance scores for the words in the input sentences for RE. Second, we propose to incorporate the overall sentence representation vectors into the cells of ON-LSTM, allowing it to compute the model-based importance scores more effectively. We also devise a novel mechanism to project the syntactic information into the computation of ON-LSTM via promoting the consistency between the syntax-based and model-based importance scores. Finally, we present a novel inductive bias for the deep learning models that exploits the similarity of the representation vectors for the
whole input sentences and the shortest dependency paths between the two entity mentions for RE. Extensive experiments are conducted to demonstrate the benefits of the proposed model. We achieve the state-of-the-art performance on three datasets for RE. In the future, we plan to apply CEON-LSTM to other related NLP tasks (e.g., Event Extraction, Semantic Role Labeling) (Nguyen et al., 2016a; Nguyen and Grishman, 2018a).
This research has been supported in part by Vingroup Innovation Foundation (VINIF) in project code VINIF.2019.DA18, the NSF grant CNS-1747798 to the IUCRC Center for Big Learning, and a gift from Adobe Research. This research is also based upon work supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via IARPA Contract No. 2019-19051600006 under the Better Extraction from Text Towards Enhanced Retrieval (BETTER) Program. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNl, IARPA, the Department of Defense, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein. This document does not contain technology or technical data controlled under either the U.S. International Traffic in Arms Regulations or the U.S. Export Administration Regulations.
Razvan C Bunescu and Raymond J Mooney. 2005. A shortest path dependency kernel for relation extraction. In EMNLP.
Yee S. Chan and Dan Roth. 2010. Exploiting background knowledge for relation extraction. In COLING.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In *NAACL-HLT*.
Lisheng Fu, Thien Huu Nguyen, Bonan Min, and Ralph Grishman. 2017. Domain adaptation for relation extraction with domain adversarial neural network. In IJCNLP.
8029
Zhijiang Guo, Yan Zhang, and Wei Lu. 2019. Attention guided graph convolutional networks for relation extraction. In ACL.
Braden Hancock, Martin Bringmann, Paroma Varma, Percy Liang, Stephanie Wang, and Christopher Ré. 2018. Training classifiers with natural language explanations. In ACL.
Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid O Seaghda, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2010. Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. In Proceedings of SEW-2009.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Yang Liu, Furu Wei, Sujian Li, Heng Ji, Ming Zhou, and Houfeng Wang. 2015. A dependency-based neural network for relation classification. In ACL.
Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In EMNLP.
Makoto Miwa and Mohit Bansal. 2016. End-to-end relation extraction using lstms on sequences and tree structures. ACL.
Minh Nguyen and Thien Huu Nguyen. 2018b. Who is killed by police: Introducing supervised attention for hierarchical lstms. In COLING.
Thien Huu Nguyen, Kyunghyun Cho, and Ralph Grishman. 2016a. Joint event extraction via recurrent neural networks. In *NAACL*.
Thien Huu Nguyen and Ralph Grishman. 2014. Employing word representations and regularization for domain adaptation of relation extraction. In ACL.
Thien Huu Nguyen and Ralph Grishman. 2015a. Relation extraction: Perspective from convolutional neural networks. In Proceedings of the 1st NAACL Workshop on Vector Space Modeling for NLP (VSM).
Thien Huu Nguyen and Ralph Grishman. 2016. Combining neural networks and log-linear models to improve relation extraction. Proceedings of IJCAI Workshop on Deep Learning for Artificial Intelligence.
Thien Huu Nguyen and Ralph Grishman. 2018a. Graph convolutional networks with argument-aware pooling for event detection. In AAAI.
Thien Huu Nguyen, Barbara Plank, and Ralph Grishman. 2015c. Semantic representations for domain adaptation: A case study on the tree kernel-based method for relation extraction. In ACL-IJCNLP.
Trung Minh Nguyen and Thien Huu Nguyen. 2019. One for all: Neural joint modeling of entities and events. In AAAI.
Tuan Ngo Nguyen, Franck Dernoncourt, and Thien Huu Nguyen. 2019a. On the effectiveness of the pooling methods for biomedical relation extraction with deep learning. In Proceedings of the Tenth International Workshop on Health Text Mining and Information Analysis (LOUHI 2019).
Nanyun Peng, Hoifung Poon, Chris Quirk, Kristina Toutanova, and Wen-tau Yih. 2017. Cross-sentence n-ary relation extraction with graph LSTMs. Transactions of the Association for Computational Linguistics, 5:101-115.
Cicero dos Santos, Bing Xiang, and Bowen Zhou. 2015. Classifying relations by ranking with convolutional neural networks. In ACL.
Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville. 2019. Ordered neurons: Integrating tree structures into recurrent neural networks. In ICLR.
Ge Shi, Chong Feng, Lifu Huang, Boliang Zhang, Heng Ji, Lejian Liao, and Heyan Huang. 2018. Genre separation network with adversarial training for cross-genre relation extraction. In EMNLP.
Linfeng Song, Yue Zhang, Daniel Gildea, Mo Yu, Zhiguo Wang, and jinsong su. 2019. Leveraging dependency forest for neural medical relation extraction. In EMNLP-IJCNLP.
Ang Sun, Ralph Grishman, and Satoshi Sekine. 2011. Semi-supervised relation extraction with large-scale word clustering. In ACL.
Kai Sheng Tai, Richard Socher, and Christopher D Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In ACL.
Van-Hien Tran, Van-Thuy Phi, Hiroyuki Shindo, and Yuji Matsumoto. 2019. Relation classification using segment-level attention-based cnn and dependency-based rnn. In NAACL-HLT.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS.
Patrick Verga, Emma Strubell, and Andrew McCallum. 2018. Simultaneously self-attending to all mentions for full-abstract biological relation extraction. In EMNLP.
Amir Pouran Ben Veyseh, Thien Huu Nguyen, and Dejing Dou. 2019. Improving cross-domain performance for relation extraction via dependency prediction and information flow control. In *IJCAI*.
8030