SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 2: F1 scores (%) of Relation Extraction on the test set of TACRED, SemEval, and TACREV. The best results in each column are highlighted in bold font.
Method	Complexity	Time	F1 (%)
IRERoBERTa (Zhou and Chen, 2021)	O(B)	7492s	74.6
IRERoBERTa + GNN	O(N)	N.A.	N.A.
IRERoBERTa + GRAPHCACHE (ours)	O(B)	7681s	75.5
Table 3: Training time, the time complexity per training step, and F1 scores of  $\mathrm{IRE}_{\mathrm{RoBERTa}}$  with our proposed message passing implemented as GNN and GRAPH-CACHE on TACRED. The training time of  $\mathrm{IRE}_{\mathrm{RoBERTa}}$  with the classical GNN is unavailable due to the our-of-memory error.  $B$  and  $N$  are the batch and dataset sizes respectively.
GRAPHCACHE achieves substantial improvements for LUKE and  $\mathrm{IRE}_{\mathrm{RoBERTa}}$  and enables them to outperform the baseline methods.
Note that LUKE and  $\mathrm{IRE}_{\mathrm{RoBERTa}}$  are both based on large pre-trained models, which have sufficiently large learning capacity to encode the individual instances. In this case, our GRAPHCACHE still improves their effectiveness by a large margin, which validates the benefits of modeling the properties: entity types and contextual topics, globally from the whole dataset. This is due to the use of the global property representations that enrich the semantics of each instance, which effectively act as prior knowledge that helps identify the relations and complements the sentence-level features.
4.3 Efficiency and Effectiveness of GRAPHCACHE
As analyzed in §3.2, GRAPHCache enhances the backbone RE models without increasing their time complexity. In the experiments, we analyze the efficiency and effectiveness of GRAPHCache on the TACRED dataset, following the experimental setting of RE in §4.2.
The methods we evaluate include  $\mathrm{IRE}_{\mathrm{RoBERTa}}$ ,  $\mathrm{IRE}_{\mathrm{RoBERTa}}$  implemented with classical GNN for message passing, and  $\mathrm{IRE}_{\mathrm{RoBERTa}}$  with our GRAPHCACHE. Tab. 3 reports the performance, where 'Time' is the training time until convergence using a Linux Server with an Intel(R) Xeon(R) E5-1650 v4 @ 3.60GHz CPU and a GeForce GTX 2080 GPU.
We notice that, compared with the classical message passing of GNN, our GRAPHCACHE method significantly reduces the time complexity per training step. As a result, our GRAPHCACHE method takes significantly less training time than the classical GNN method, and exhibits similar efficiency to the original  $\mathrm{IRE}_{\mathrm{RoBERTa}}$  without message pass
Method	TACRED	TACREV
LUKE (Yamada et al., 2020)	76.5	82.9
LUKE + GRAPHCACHE (ours)	78.9	85.6
IRERoBERTa (Zhou and Chen, 2021)	78.7	86.9
IRERoBERTa + GRAPHCACHE (ours)	80.1	88.2
Table 4: Test F1 scores  $(\%)$  of Relation Extraction on the filtered test sets (see  $\S 4.4$  ), i.e., the instances containing unseen entities.
Technique	F1 (%)	Δ	Cumu Δ
LUKE (Yamada et al., 2020)	72.7	0	0
+ Entity Types	73.4	+0.7	+0.7
+ Contextual Topics	74.8	+1.4	+2.1
Table 5: Effects of different properties in our heterogeneous graph on the RE of TACRED.
ing between sentences. The running time and F1 of  $\mathrm{IRE}_{\mathrm{RoBERTa}}$  with GNN is unavailable due to the out-of-memory error. This agrees with the theoretical analysis in §3.2.  $N$  and  $B$  denote the data and batch sizes respectively.  $\mathrm{IRE}_{\mathrm{RoBERTa}}$  's time complexity is  $\mathcal{O}(B)$ , which is the same as the original RoBERTa, while the time complexity of RoBERTa with GNN is  $\mathcal{O}(N)$ , being significantly higher than our GRAPHCAcHcE. In practice,  $N$  is generally large, and  $N\gg B$ , e.g.,  $|\mathcal{E}| > 1\times 10^5$  and  $B < 100$  holds for TACRED and state-of-the-art models.
In terms of effectiveness, our GRAPHCACHE leads to substantial improvements for RoBERTa. Our GRAPHCACHE enriches the input features for RE on every sentence by utilizing the dataset-level information beyond the individual sentences. GRAPHCACHE implements the attention module to incorporate the global property features from different topic caches with adaptive weights, which capture the most relevant information for the target relation. The improvements in effectiveness are rooted in the message passing mechanism between sentences, which mines the property information beyond individual instances and acts as a complementary to the sentence-level semantics. Our GRAPHCACHE method resolves the efficiency issues of message passing based on the caching mechanism, which updates the properties' representations in an online manner.
4.4 Analysis on Unseen Entities
Some previous work (Zhang et al., 2018; Joshi et al., 2020) suggests that RE models may not gen
Figure 3: The F1 scores (\% in z-axis) of  $\mathrm{IRE}_{\mathrm{RoBERTa}}$  with GRAPHCACHED on relation extraction on TACRED with different hyper-parameters  $P$  and  $K$ .
eralize well to unseen entities. To evaluate whether the RE models can generalize to unseen entities, existing work designs a filtered evaluation setting (Zhou and Chen, 2021). This setting removes all testing instances containing entities from the training set of TACRED and TACREV, which results in filtered test sets of 4,599 instances on TACRED and TACREV. These filtered test sets only contain instances with unseen entities during training.
We present the experimental results on the filtered test sets in Tab. 4. Our GRAPHCACHE still achieves consistently substantial improvements for LUKE and  $\mathrm{IRE}_{\mathrm{RoBERTa}}$  on the TACRED and TACREV datasets. Specifically, our GRAPHCACHE improves the F1 scores of LUKE by  $3.1\%$  on TACRED,  $3.3\%$  on TACREV, and improves  $\mathrm{IRE}_{\mathrm{RoBERTa}}$  by  $1.8\%$  on TACRED,  $1.5\%$  on TACREV. Taking a closer look, we observe that the improvements given by GRAPHCACHE on the filtered test sets are generally larger than those on the original test sets. The reason is that our GRAPHCACHE mines global information from the whole dataset and uses it as the prior knowledge for RE, which is not influenced by the entity names in individual sentences. When the entity names are new to the RE models, the semantic information is relatively scarce and our mined global information plays a more important role to augment the sentence-level representations.
We investigate the contributions of properties that we consider for constructing the heterogeneous graph. We apply different kinds of properties sequentially with our GRAPHCACHE on the LUKE model. The results are presented in Tab. 5. Our entity type nodes improve the effectiveness of LUKE by modeling the entity information globally on the
Input sentence	Method	Prediction	Entity type	Topic keyword
Founded in 1947 by two brothers, Eugene and Quentin Fabris, New Fabris started out making sewing machine parts in the 1990s.	LUKE	founded ×	subject: Person object: Date	[brother, found, sister, parent, establish, machine, business, organize, instrument, make]
+ GRAPHCache	no Relation ✓
According to the suspect, Gonzalez was strangled and buried the day after the video was made, Rosas said.	LUKE	no Relation ×	subject: Person object: Date	[strangle, die, after, when, injury, day, hospital, police, murder, later]
+ GRAPHCache	date_of_death ✓
He was forced to close his bar and now works occasionally at the University of Foreigners, which Knox and Kercher attended.	LUKE	no Relation ×	subject: Person object: Organization	[university, student, attend, opening, work, school, job, professor, exchange, education]
+ GRAPHCache	schools_attended ✓
Margaret Garritsen graduated from the University of Michigan as an American Association of University scholar.	LUKE	schools_attended ×	subject: Organization object: Organization	[graduate, government, association, degree, university, technology, science, scholar, receive, research]
+ GRAPHCache	no Relation ✓
Table 6: A case study for LUKE and our GRAPHCache on the relation extraction dataset TACRED. We report the predicted relations of different methods, the entity types, and the top 10 words with the highest probabilities of the topic that the sentence attends with the highest attention weight.
dataset level to enrich the semantics of every sentence. This finding is consistent with Peng et al. (2020), suggesting that the entity information can provide richer information to improve RE. Furthermore, the contextual topics lead to more significant improvements than the entity types, since the contextual information is fundamental for identifying the relations.
Finally, we analyze the sensitivity of GRAPH-CACHE to the hyper-parameters  $K, P$ , where  $K$  is the number of topics and  $P$  is the number of relevant topics assigned to an instance. The result is visualized in Fig. 3. We vary  $K$  among  $\{10, 20, 30, 40, 50, 60\}$  and  $P$  among  $\{1, 2, 3, 4, 5, 6\}$ . The performance of  $\mathrm{IRE}_{\mathrm{RoBERTa}}$  with GRAPH-CACHE is relatively smooth when parameters are within certain ranges. However, extremely small values of  $K$  and large  $P$  result in poor performances. Too small  $K$  cannot effectively model the complex contextual topics in the large text corpus, while too large  $P$  induces irrelevant or noisy features for every instance. Moreover, only a poorly set hyper-parameter does not lead to significant performance degradation, which demonstrates that our GRAPH-CACHE framework is able to effectively mine the beneficial properties at the dataset level and use them to enhance the relation representations for RE.
We conduct a case study to investigate the effects of our GRAPHCACHE. Tab. 6 gives a qualitative comparison example between LUKE and the LUKE with our GRAPHCACHE on the relation extraction dataset TACRED. The result shows that the global property information that we mine from the whole dataset can guide the RE systems to make correct predictions. For example, in the first row, we model the global entity type information of the subject as the person and the object as the date from the whole dataset. This type information acts as the prior knowledge that prevents the model from making the wrong relation prediction of 'founded' between the entities 'Quentin Fabris' and '1947' (date). Similarly, in the final row, our GRAPHCACHE filters out the incorrect relation 'schools Attend', since we model the entity type information from the whole dataset and thus enable the model to be aware that this relation cannot hold for the subject type as 'organization'.
In addition, in the second row, the sentence 'According to the suspect, Gonzalez was strangled and buried the day after the video was made, Rosas said.' attends to the topic of keywords '[strangle, die, after, when, injury, day, hospital, police, murder, later]' in our heterogeneous graph, which enriches the semantics of the sentence with the
context related to the death and time. This helps the model to make the correct relation prediction 'date_of_death'.
In this paper, we study the efficient message passing to enhance the relation extraction models. We propose a novel method named GRAPHCACHE, which provides efficient message passing between instances in the whole dataset. GRAPHCACHE is a model-agnostic technique that can be incorporated into popular relation extraction models to enhance their effectiveness without increasing their time complexity. In our work, we present a simple yet effective implementation of GRAPHCACHE, which models two universal and essential properties for relation extraction: entity information and textual context. Our experimental results show that GRAPHCACHE, with our heterogeneous graph, yields significant gains for the sentence-level relation extraction in an efficient manner.
The authors would like to thank the anonymous reviewers for their discussion and feedback.
Muhao Chen and Wenxuan Zhou are supported by the National Science Foundation of United States Grant IIS 2105329, and by the DARPA MCS program under Contract No. N660011924033 with the United States Office Of Naval Research. Except for Muhao Chen and Wenxuan Zhou, this paper is supported by NUS ODPRT Grant R252-000-A81-133 and Singapore Ministry of Education Academic Research Fund Tier 3 under MOEs official grant number MOE2017-T3-1-007.
Christoph Alt, Aleksandra Gabryszak, and Leonhard Hennig. 2020a. TACRED revisited: A thorough evaluation of the TACRED relation extraction task. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1558-1569, Online. Association for Computational Linguistics.
Christoph Alt, Aleksandra Gabryszak, and Leonhard Hennig. 2020b. TACRED revisited: A thorough evaluation of the TACRED relation extraction task. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1558-1569, Online. Association for Computational Linguistics.
Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019. Matching the blanks: Distributional similarity for relation learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2895-2905, Florence, Italy. Association for Computational Linguistics.
Jasmijn Bastings, Ivan Titov, Wilker Aziz, Diego Marcheggiani, and Khalil Sima'an. 2017. Graph convolutional encoders for syntax-aware neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1957-1967, Copenhagen, Denmark. Association for Computational Linguistics.
David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. the Journal of machine Learning research, 3:993-1022.
Fenia Christopoulou, Makoto Miwa, and Sophia Ananiadou. 2019. Connecting the dots: Document-level neural relation extraction with edge-oriented graphs. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4925-4936, Hong Kong, China. Association for Computational Linguistics.
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. 2016. Convolutional neural networks on graphs with fast localized spectral filtering. Advances in neural information processing systems, 29:3844-3852.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Bayu Distiawan, Gerhard Weikum, Jianzhong Qi, and Rui Zhang. 2019. Neural relation extraction for knowledge base enrichment. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 229-240.
Zhijiang Guo, Yan Zhang, and Wei Lu. 2019a. Attention guided graph convolutional networks for relation extraction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 241-251, Florence, Italy. Association for Computational Linguistics.
Zhijiang Guo, Yan Zhang, and Wei Lu. 2019b. Attention guided graph convolutional networks for relation extraction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 241-251, Florence, Italy. Association for Computational Linguistics.
William L Hamilton, Rex Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 1025-1035.
Mikael Henaff, Joan Bruna, and Yann LeCun. 2015. Deep convolutional networks on graph-structured data. arXiv preprint arXiv:1506.05163.
Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid O Seaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2019. Semeval-2010 task 8: Multiway classification of semantic relations between pairs of nominals. arXiv preprint arXiv:1911.10422.
Hamed Jelodar, Yongli Wang, Chi Yuan, Xia Feng, Xiahui Jiang, Yanchao Li, and Liang Zhao. 2019. Latent dirichlet allocation (lda) and topic modeling: models, applications, a survey. Multimedia Tools and Applications, 78(11):15169-15211.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. Span-BERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77.
Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Shengfei Lyu and Huanhuan Chen. 2021. Relation classification with entity type restriction. arXiv preprint arXiv:2105.08393.
Shie Mannor, Dori Peleg, and Reuven Rubinstein. 2005. The cross entropy method for classification. In Proceedings of the 22nd international conference on Machine learning, pages 561-568.
Diego Marcheggiani and Ivan Titov. 2017. Encoding sentences with graph convolutional networks for semantic role labeling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1506-1515, Copenhagen, Denmark. Association for Computational Linguistics.
Guoshun Nan, Zhijiang Guo, Ivan Sekulic, and Wei Lu. 2020. Reasoning with latent structure refinement for document-level relation extraction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1546-1557, Online. Association for Computational Linguistics.
Hao Peng, Tianyu Gao, Xu Han, Yankai Lin, Peng Li, Zhiyuan Liu, Maosong Sun, and Jie Zhou. 2020. Learning from context or names? an empirical
study on neural relation extraction. arXiv preprint arXiv:2010.01923.
Hao Peng, Jianxin Li, Yu He, Yaopeng Liu, Mengjiao Bao, Lihong Wang, Yangqiu Song, and Qiang Yang. 2018. Large-scale hierarchical text classification with recursively regularized deep graph-cnn. In Proceedings of the 2018 world wide web conference, pages 1063-1072.
Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019. Knowledge enhanced contextual word representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 43-54, Hong Kong, China. Association for Computational Linguistics.
Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. 2017. Pointnet: Deep learning on point sets for 3d classification and segmentation.
Sunil Kumar Sahu, Fenia Christopoulou, Makoto Miwa, and Sophia Ananiadou. 2019. Inter-sentence relation extraction with document-level graph convolutional neural network. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4309-4316, Florence, Italy. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Haoyu Wang, Muhao Chen, Hongming Zhang, and Dan Roth. 2020a. Joint constrained learning for event-event relation extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 696-706.
Linlin Wang, Zhu Cao, Gerard de Melo, and Zhiyuan Liu. 2016. Relation classification via multi-level attention CNNs. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1298-1307, Berlin, Germany. Association for Computational Linguistics.
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Guihong Cao, Daxin Jiang, Ming Zhou, et al. 2020b. K-adapter: Infusing knowledge into pre-trained models with adapters. arXiv preprint arXiv:2002.01808.
Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. 2019. Simplifying graph convolutional networks. In International conference on machine learning, pages 6861-6871. PMLR.
Lingfei Wu, Yu Chen, Kai Shen, Xiaojie Guo, Hanning Gao, Shucheng Li, Jian Pei, and Bo Long. 2021. Graph neural networks for natural language processing: A survey. arXiv preprint arXiv:2106.06090.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. 2018. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826.
Fuzhao Xue, Aixin Sun, Hao Zhang, and Eng Siong Chng. 2021. Gdpnet: Refining latent multi-view graph for relation extraction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14194-14202.
Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020. LUKE: Deep contextualized entity representations with entity-aware self-attention. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6442–6454, Online. Association for Computational Linguistics.
Liang Yao, Chengsheng Mao, and Yuan Luo. 2019. Graph convolutional networks for text classification. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 7370-7377.
Dian Yu, Kai Sun, Claire Cardie, and Dong Yu. 2020. Dialogue-based relation extraction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4927-4940.
Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao. 2014. Relation classification via convolutional deep neural network. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 2335-2344, Dublin, Ireland. Dublin City University and Association for Computational Linguistics.
Jichuan Zeng, Jing Li, Yan Song, Cuiyun Gao, Michael R. Lyu, and Irwin King. 2018. Topic memory networks for short text classification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3120-3131, Brussels, Belgium. Association for Computational Linguistics.
Shuang Zeng, Runxin Xu, Baobao Chang, and Lei Li. 2020. Double graph based reasoning for document-level relation extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1630-1640, Online. Association for Computational Linguistics.
Yuhao Zhang, Peng Qi, and Christopher D. Manning. 2018. Graph convolution over pruned dependency trees improves relation extraction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2205-2215, Brussels, Belgium. Association for Computational Linguistics.
Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Position-aware attention and supervised data improve slot filling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP 2017), pages 35-45.
Xinyan Zhao, Feng Xiao, Haoming Zhong, Jun Yao, and Huanhuan Chen. 2020. Condition aware and revise transformer for question answering. In Proceedings of The Web Conference 2020, pages 2377-2387.
Wenxuan Zhou and Muhao Chen. 2021. An improved baseline for sentence-level relation extraction. arXiv preprint arXiv:2102.01373.
Hao Zhu, Yankai Lin, Zhiyuan Liu, Jie Fu, Tat-Seng Chua, and Maosong Sun. 2019. Graph neural networks with generated parameters for relation extraction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1331-1339, Florence, Italy. Association for Computational Linguistics.