SlowGuess/NAACL_2022_OCR huggingface dataset:
instance label independently, thus unable to exploit the label dependency. (Lin et al., 2020) and (Zhang and Ji, 2021) resort to beam search that step by step constructs a complete decoding assignment  $Y$  for the instances in  $V$  by expanding an initially empty assignment. Each step corresponds to an instance in  $V$  where only top candidate labels for the instance are considered for assignment expansion and only top partial assignments produced so far are kept for the next step. Unfortunately, the selection of top candidate labels for expansion at each step is based only on the local scores  $\mathbf{s}_i$ , which might discard the candidates that can eventually provide greater global scores. To overcome this issue, we propose to apply Simulated Annealing (SA) (Kirkpatrick et al., 1983) to search for the optimal assignment  $\hat{Y}$  for  $V$ . SA is a probabilistic algorithm that is able to approximately find the global optimum of a function (Kirkpatrick et al., 1983; Van Laarhoven and Aarts, 1987). Algorithm 1 presents our implementation for SA to find  $\hat{Y}$ .
Datasets	Split	#sents	#ents	#rels	#events
ACE05-R	Train	10,051	26,473	4,788	-
Dev	2,424	6,362	1,131	-
Test	2,050	5,476	1,151	-
ACE05-E	Train	17,172	29,006	4,664	4,202
Dev	923	2,451	560	450
Test	832	3,017	636	403
ACE05-E+	Train	19,240	47,525	7,152	4,419
Dev	902	3,422	728	468
Test	676	3,673	802	424
ERE-EN	Train	14,219	38,864	5,045	6,419
Dev	1,162	3,320	424	552
Test	1,129	3,291	477	559
ERE-ES	Train	7,067	11,839	1,698	3,272
Dev	556	886	120	210
Test	546	811	108	269
Table 1: Data statistics. #sents, #ent, #rels, and #events indicate the number of sentences, entity mentions, relations, and events respectively.
The input for the algorithm is the initial configuration  $\hat{Y}_{cur} = \hat{Y}_0 = \{\hat{y}_{i,0}\}$ , which contains the greedily predicted labels for each instance:  $\hat{y}_{i,0} = \operatorname{argmax}_{c\in C}\mathbf{s}_i[c]$ . The algorithm then runs over  $N_{iter}$  iterations to improve the global score  $s(\hat{Y}_{cur})$  for the current label configuration  $\hat{Y}_{cur}$ . This is done via updating the current configuration to a successor configuration  $\hat{Y}_{new}$  that gives a higher global score (i.e.,  $\delta_n > 0$ ). A successor configuration is obtained via the function random_successor() by randomly changing some label  $\hat{y}_i\in \hat{Y}_{cur}$ . Different from beam search decoding with partial assignments, each searching step in SA examines a complete label assignment for the instances in  $V$  to provide complete information to measure the
global scores/quality of the assignments. Importantly, SA sometimes allows the current configuration to transition to a successor configuration with a lower global score (i.e.,  $\delta_{n} \leq 0$ ) with an acceptance probability of  $p = \exp \left( \frac{\delta_{n}}{t} \right)$ . Here,  $t$  is the temperature of the algorithm, gradually decreased via  $t \gets T / n$  ( $T$  is a hyper-parameter). This exploration property enables SA to escape from local optimum configurations, thus increasing the chance to find the globally optimal configuration  $\hat{Y}$ .
Datasets: Following previous work (Wadden et al., 2019; Lin et al., 2020; Zhang and Ji, 2021; Nguyen et al., 2021a; Lu et al., 2021; Hsu et al., 2021), we conduct experiments on 5 different datasets created by the 2005 Automatic Content Extraction (ACE05) (Walker et al., 2006) and Entity Relation Event (ERE) (Song et al., 2015) programs. The three ACE05 datasets feature ACE05-R, ACE05-E, and ACE-E+, all in English, involving 33 event types, 7 entity types, 6 relation types, and 22 argument roles. The two ERE datasets are ERE-EN (English portion) and ERE-ES (Spanish portion), introducing 38 event types, 7 entity types, 5 relation types, and 20 argument roles. We use the same data processing and train/dev/test splits as the prior work for a fair comparison. Detailed statistics for the datasets are shown in Table 1.
Baselines: We compare our method, called GraphIE, with the following baselines for JointIE:
Generative baselines: Text2event (Lu et al., 2021) and DEGREE (Hsu et al., 2021). The generative baselines perform ETD and EAE via formulating the tasks as text generation. The models receive an input sentence and generate an output text containing text spans and labels for event triggers and event arguments, structured in a way that a post-processing step can be used to extract ETD and EAE predictions for the models.
Classification baselines: OneIE (Lin et al., 2020), AMRIE (Zhang and Ji, 2021), and FourIE (Nguyen et al., 2021a). The classification baselines represent the instances for ETD, EMR, EAE, and RE via a shared encoder and perform classification for the instances based on task-specific label distributions. AMRIE and FourIE employ a heuristic dependency graph among task instances to improve representation learning. Dependency between instance labels is exploited in OneIE and AMRIE via a beam search decoding with manually-designed
PLMs	Model	ACE05-R	ACE05-E	ACE05-E+	ERE-EN	ERE-ES
Ent	Rel	Ent	Trg	Arg	Ent	Rel	Trg	Arg	Ent	Rel	Trg	Arg	Ent	Rel	Trg	Arg
T5	Text2event	-	-	-	71.9	53.8	-	-	71.8	54.4	-	-	59.4	48.3	-	-	-	-
BART	DEGREE	-	-	-	72.2	56.0	-	-	71.7	58.0	-	-	56.6	51.1	-	-	-	-
BERT	OneIE	88.6	63.4	90.2	74.7	56.8	89.6	58.6	72.8	54.8	87.0	53.2	57.0	46.5	81.3	48.1	56.8	40.3
AMRIE*	88.7	67.2	90.8	75.3	58.2	90.4	62.9	72.8	56.3	86.9	55.5	58.3	44.2	-	-	-	-
FourIE	88.9	68.9	91.3	75.4	58.0	91.1	63.6	73.3	57.5	87.4	56.1	57.9	48.6	82.2	57.9	57.1	42.3
GraphIE	88.9	69.5	90.6	75.7	58.8	91.0	65.4	74.8	59.9	87.2	57.8	61.4	52.2	81.4	58.9	61.3	45.7
RoBERTa	OneIE*	89.0	65.2	90.2	74.7	55.6	90.8	60.4	72.5	56.3	86.3	52.8	57.1	47.1	83.7	57.5	58.3	42.5
AMRIE	89.2*	66.8*	92.1	75.0	58.6	91.0*	62.8*	72.7*	57.7*	87.9	55.2	61.4	45.0	-	-	-	-
FourIE*	89.1	67.5	91.6	74.9	58.7	91.1	63.1	72.8	58.3	88.0	56.2	61.5	49.1	83.9	61.0	62.3	44.2
GraphIE	89.3	68.5	91.4	75.1	59.4	91.6	66.0	73.3	60.2	87.7	57.0	62.0	54.7	84.3	62.3	65.7	46.9
global features, and in FourIE via global type dependency regularization. FourIE and AMRIE are the current state-of-the-art models for JointIE.
Hyper-parameters: Prior work for JointIE employs two different versions of pre-trained language models (PLM), i.e., BERT (Devlin et al., 2019; Lin et al., 2020; Nguyen et al., 2021a) and RoBERTa (Liu et al., 2019; Zhang and Ji, 2021), which might cause incompatible compassion. To this end, we explore both BERT and RoBERTa to obtain the word representations  $\mathbf{x}_i$  for GraphIE for a fair comparison. For the Spanish ERE-ES dataset, following prior work (Lin et al., 2020; Nguyen et al., 2021a), we utilize the multilingual versions of BERT and RoBERTa. For each PLM, we fine-tune the hyperparameter for GraphIE on the development data.
In particular, the best values for the hyperparameters of the proposed model are reported as follows. We employ the learning rate of  $1e - 5$  for the models with the BERT-based PLM (i.e., using bert-large-cased and bert-multilingual-cased) and the learning rate of  $5e - 6$  for the RoBERTa-based PLM (i.e., using roberta-large and xlm-roberta-large). For other hyper-parameters, our tuning process results in the same values for BERT-based and RoBERT-based models: Adam (Kingma and Ba, 2014) for the optimizer, batch size of 10, 100 for the size of the dependency relation embeddings, 400 for the size of the hidden vector for the feedforward networks, 200 for the hidden vector size in the GCN model, 2 for the number of layers for the feed-forward networks and GCN model,  $\gamma = 1$  for the trade-off hyper-parameter for the global score,  $N_{\text{noi}} = 5$  for the number of noise examples for the contrastive loss (we re-sample the noise examples every epoch),  $T = 5$  for the initial temperature,  $N_{\text{iter}} = 50$  for the number of iterations of Simulated Annealing (SA), and  $\epsilon = 0.1$  for the temperature threshold for the SA decoding.
Comparison with Baselines: We compare the proposed model GraphIE with the baselines on test data of the 5 datasets in Table 2. As can be seen, the generative baselines perform worse than the classification models on most of the settings. This might be due the implicit modeling of the label distributions and the assumption of a decoding order for task instances that limit the interactions of instance labels. Comparing OneIE, FourIE and AMRIE, it is clear that the exploitation of instance and label dependency in the training phase in FourIE can lead to better performance for JointIE than using such dependency in the decoding phase as done by OneIE and AMRIE over most tasks and PLMs. Most importantly, the proposed GraphIE significantly outperforms all the baselines across a majority of settings for tasks, datasets and PLMs, thus demonstrating the benefits of induced dependency graph, joint label distribution estimation, and simulated annealing for decoding in our method.
Table 2: Model performance on the test data of 5 datasets. "Ent", "Rel", "Trg", and "Arg" are the F1 scores for identification and classification of entity mentions, event triggers, relations, and event arguments respectively. * indicates results that are not reported in the original papers but produced by their official code. Underlined numbers designate the tasks where GraphIE is significantly better  $(\mathrm{p} < 0.01)$  than the baselines.
Model (all use Roberta)	ACE05-E+
Ent	Rel	Trg	Arg
GraphIE	89.8	67.2	72.6	66.3
- induced dep	89.3	65.8	71.3	65.0
- semantic-based dep	89.0	66.4	71.6	65.9
- syntactic-based dep	89.4	66.3	72.0	65.4
- induced dep + heuristic dep	89.3	66.2	71.7	65.5
- GCN	89.4	65.6	70.9	64.6
Table 3: Performance (F1) on the ACE05-E+ development data.
Ablation Study: To understand the contributions of each proposed component to GraphIE, we conduct ablation experiments where we remove each component from the full model and evaluate the performance of the remaining models.
The first three ablated models in Table 3 are "induced dep", "semantic dep", and "syntactic dep", formed by excluding the dependency weight induction of  $\alpha_{ij}$  (i.e., setting  $\alpha_{ij} = 1$ ),
the semantic-based dependency  $\alpha_{ij}^{sem}$ , and the syntactic-based dependency  $\alpha_{ij}^{syn}$  (respectively) from the model computation. In each case, the performance of GraphIE decreases significantly; the removal of both semantic- and syntactic-based dependency in “-induced dep” leads to the largest performance drop. This shows that the semantic and syntactic weighting captures complementary information for instance dependency induction that is useful for our model. The next ablated model “-induced dep + heuristic dep” is obtained by replacing the induced dependency graph represented by  $\alpha_{ij}$  with the heuristic dependency graph for instances from the best baseline FourIE. The decrease in the performance of this model suggests that the induced dependency graph is better than the heuristic graph for JointIE. The final ablated model “-GCN” in Table 3 eliminates the GCN component from our full model. The result shows that GCN is beneficial to exploit the induced dependency graph to improve representation learning.
Model (all use Roberta)	ACE05-E+
Ent	Rel	Trg	Arg
GraphIE	89.8	67.2	72.6	66.3
- joint distribution	89.3	65.5	70.9	64.5
- SA + greedy	89.2	65.9	71.2	65.2
- SA + beam	89.5	66.0	71.5	65.4
- SA + hill climbing	89.5	66.8	71.7	65.3
OneIE	88.7	64.2	69.5	63.2
- beam + SA	88.1	63.9	69.1	62.7
AMRIE	89.4	65.4	71.2	64.4
- beam + SA	88.8	65.1	70.5	64.1
In Table 4, we first eliminate the computation of the joint label distribution  $P(Y|\mathbf{w},V)$  from GraphIE. As such, the “-joint distribution” model employs the local label distributions  $P(y_{i}|\mathbf{w},V)$  to train models and infer labels (with greedy decoding). Due to the significantly worse performance of “-joint distribution”, it is clear that directly estimating the joint label distribution is helpful for JointIE. To evaluate the benefit of the proposed SA, we replace it with other decoding algorithms for GraphIE, including greedy search, beam search and hill climbing. The beam search is implemented with our global score function  $s(Y)$  and follows those in (Lin et al., 2020; Zhang and Ji, 2021) while hill climbing is implemented by removing the configuration exploration in lines 11-12 of Algorithm 1. As reported in Table 4, SA performs much better than other decoding algorithms for GraphIE, thus
demonstrating SA's ability to find globally optimal labels. In addition, we also attempt to replace the beam search decoding in OneIE and AMRIE with SA, which indeed leads to worse performance for such models as shown in the last four rows of Table 4. We attribute this to the learning of the global scores for configurations in OneIE and AMRIE that involves a limited set of predefined global features. Such features do not exist for many possible assignments  $Y$  for  $V$ , thus causing poor global score computation and hindering the configuration ranking critically required by SA.
Table 4: Performance (F1) on the ACE05-E+ development data.
Label pair	Transition score
Argument:Origin, Argument:Place)	10.02
(Event:Transport, Relation:Physical)	4.33
(Relation:Org-Aff, Relation:Part-Whole)	3.58
(Event:Execute, Event:Sentence)	2.58
(Event:Die, Event:Be-Born)	-2.34
(Event:Attack, Argument:Origin)	-87.07
(Relation:Per-Soc, Entity:Facility)	-93.93
(Transport, Attacker)	-99.91
Table 5: Transition scores for some label pairs learned by our model on ACE05-E+.
Analysis: To further understand the advantages of GraphIE over baseline models, we manually analyze the instances on the ACE05-E+ development data where GraphIE can make correct predictions, but the best baseline model FourIE fails. Figure 2 presents some instances along with their edges and weights in the dependency graphs. The most important insight from our analysis is that GraphIE is able to connect an instance (e.g., blew) with other supporting instances (e.g., suicide) in the dependency graph to provide vital information to facilitate correct prediction. Such supporting instances do not share any event trigger or entity mention with the current instance that cannot establish links in FourIE and lead to failure predictions.
Finally, Table 5 shows the transition scores  $\pi_{y_i\leftrightarrow y_j}$  learned by GraphIE for some label pairs in ACE05-E+. The table show that our model is able to learn high scores for correlated label pairs (e.g., the Execute and Sentence event types) and very low scores for uncorrelated label pairs (e.g., an argument for a Transport event cannot play the role Attacker).
Capturing dependency between IE tasks has been a main focus of previous work on Joint IE. Early work employed feature engineering methods (Roth and Yih, 2004; Yu and Lam, 2010; Li et al., 2013;
Example	GraphIE	FourIE
In the January attack, two Palestinian suicide bombers blew themselves up in central Tel Aviv, killing 23 other people.
Explanation: “blew” is correctly predicted by GraphIE as a “Die” event trigger while FourIE incorrectly predicted it as an “Attack” event trigger.	Event: Die
(blew, bombers)
blew
0.56
(blew, themselves)
suicide	Event: Attack
1.0
(blew, bombers)
blew
1.0
(blew, themselves)
suicide
We pretty much know that Marinello, while on the board, has arranged to get future money from the USCF.
Explanation: The relation between “Marinello” and “USCF” is correctly predicted by GraphIE as a “ORG-AFF” relation while FourIE incorrectly predicted it as a “GEN-AFF” relation.	Relation: ORG-AFF
(Marinello, USCF)
0.86
0.61
0.85
USCF
board	Relation: GEN-AFF
(Marinello, USCF)
1.0
1.0
USCF
board
A second rocket landed in farmlands and the other hit a house inside the refugee camp, ...
Explanation: “other” is correctly predicted by GraphIE as an “Instrument” for the event trigger “hit” while FourIE incorrectly predicted it as an “Attacker” for the event trigger “hit”.	Event Argument: Instrument
(hit, other)
0.64
0.82
0.75
rocket	Event Argument: Attacker
(hit, other)
1.0
1.0
rocket
Figure 2: Instances along with their dependency subgraphs in ACE05-E+. Supporting instances are underlined.
Yang and Mitchell, 2016). Later work applied deep learning via shared parameters to facilitate joint modeling for IE, however, for only two or three tasks (Nguyen et al., 2016a; Zheng et al., 2017; Bekoulis et al., 2018; Luan et al., 2019; Zhang et al., 2019; Nguyen and Nguyen, 2019). Recently, the four IE tasks have been solved jointly (Wadden et al., 2019; Lin et al., 2020; Zhang and Ji, 2021; Paolini et al., 2021; Lu et al., 2021; Nguyen et al., 2021a). However, such recent works only employ heuristics to manually design dependency graphs for instances. Mean-field factorization of the joint label distribution for JointIE instances is dominant in prior work.
Our work is also related to prior work that uses CRFs (Lafferty et al., 2001; Chiu and Nichols, 2016) to estimate joint distribution of instance labels. Sequence labeling is a typical problem that has been solved by CRFs, including part of speech tagging and named entity recognition (Lafferty et al., 2001; Ekbal et al., 2007; Shishtla et al., 2008; Sobhana et al., 2010; Zea et al., 2016; Chiu and Nichols, 2016; Xu et al., 2017). However, these prior work only employ CRFs for simple graph structures (i.e., linear chains). A few prior work has considered CRFs for more complicated graph structures (Sun et al., 2017; Gao et al., 2019; Qu et al., 2019; Yuan and Ji, 2020); however, none of such works has applied CRFs for JointIE as we do.
We propose a novel model for jointly solving four IE tasks (EMR, ETD, EAE, and RE). Our proposed model learns a dependency graph among the in
stances of the tasks via a novel edge weighting mechanism. We also estimate the joint distribution among instance labels to fully enable interactions between instance labels for improved performance. The experimental results show that our model achieves best performance for multiple JointIE tasks across 5 datasets and 2 languages. In the future, we plan to extend our method to cover more IE tasks such as event coreference resolution.
This research has been supported by the Army Research Office (ARO) grant W911NF-21-1-0112 and the NSF grant CNS-1747798 to the IUCRC Center for Big Learning. This research is also based upon work supported by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via IARPA Contract No. 2019-19051600006 under the Better Extraction from Text Towards Enhanced Retrieval (BETTER) Program. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ARO, ODNI, IARPA, the Department of Defense, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein. This document does not contain technology or technical data controlled under either the U.S. International Traffic in Arms Regulations or the U.S. Export Administration Regulations.
Giannis Bekoulis, Johannes Deleu, Thomas Demeester, and Chris Develder. 2018. Adversarial training for multi-context joint entity and relation extraction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
Yubo Chen, Liheng Xu, Kang Liu, Daojian Zeng, and Jun Zhao. 2015. Event extraction via dynamic multipooling convolutional neural networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing.
Jason PC Chiu and Eric Nichols. 2016. Named entity recognition with bidirectional LSTM-cnns. In Transactions of the Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Xinya Du and Claire Cardie. 2020. Event extraction by answering (almost) natural questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 671-683, Online. Association for Computational Linguistics.
Asif Ekbal, Rejwanul Haque, and Sivaji Bandyopadhyay. 2007. Bengali part of speech tagging using conditional random field. In Proceedings of the seventh International Symposium on Natural Language Processing, SNLP-2007.
G David Forney. 1973. The viterbi algorithm. Proceedings of the IEEE, 61(3):268-278.
Hongchang Gao, Jian Pei, and Heng Huang. 2019. Conditional random field enhanced graph convolutional neural networks. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 276-284.
Michael U Gutmann and Aapo Hyvarinen. 2012. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. Journal of Machine Learning Research, 13(2).
I Hsu, Kuan-Hao Huang, Elizabeth Boschee, Scott Miller, Prem Natarajan, Kai-Wei Chang, Nanyun Peng, et al. 2021. Degree: A data-efficient generative event extraction model. arXiv preprint arXiv:2108.12724.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In Proceedings of the 5th International Conference on Learning Representations.
Scott Kirkpatrick, C Daniel Gelatt, and Mario P Vecchi. 1983. Optimization by simulated annealing. science, 220(4598):671-680.
John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data.
Viet Dac Lai, Tuan Ngo Nguyen, and Thien Huu Nguyen. 2020. Event detection: Gate diversity and syntactic importance scores for graph convolution neural networks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Fayuan Li, Weihua Peng, Yuguang Chen, Quan Wang, Lu Pan, Yajuan Lyu, and Yong Zhu. 2020. Event extraction as multi-turn question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 829-838.
Qi Li, Heng Ji, and Liang Huang. 2013. Joint event extraction via structured prediction with global features. In Proceedings of the 51th Annual Meeting of the Association for Computational Linguistics.
Ying Lin, Heng Ji, Fei Huang, and Lingfei Wu. 2020. A joint neural model for information extraction with global features. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Yaojie Lu, Hongyu Lin, Jin Xu, Xianpei Han, Jialong Tang, Annan Li, Le Sun, Meng Liao, and Shaoyi Chen. 2021. Text2event: Controllable sequence-to-structure generation for end-to-end event extraction. arXiv preprint arXiv:2106.09232.
Yi Luan, Dave Wadden, Luheng He, Amy Shah, Mari Ostendorf, and Hannaneh Hajishirzi. 2019. A general framework for information extraction using dynamic span graphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association