SlowGuess/ACL_2020_OCR huggingface dataset:
CoNLL 2012 SRL As our model bears similarities to the SRL models proposed by He et al. (2018) and Ouchi et al. (2018), we evaluate our model on the sentence-level CoNLL 2012 dataset as a sanity check. Based on a small hyperparameter sweep, our model achieves  $81.4\mathrm{F}_1$  when given gold predicate spans and  $81.2\mathrm{F}_1$  when not given gold predicates. $^{13}$  Our model's recall is harmed because our span pruning occurs at the document level rather than at the sentence level, which leads to overpruning in some sentences. Although our model is designed to accommodate cross-sentence links, it maintains competitive performance on sentence-level SRL.
Model	Dev. F1	P	R	F1
Our model	69.9	62.8	74.9	68.3
Our modelTCD	75.1	78.1	69.2	73.3
Most common	17.3	15.7	15.7	15.7
Fixed triggerTCD	60.2	83.7	41.9	55.8
Context as triggerTCD	62.1	80.5	45.8	58.4
Distractor args	24.3	60.5	15.1	24.2
Distractor argSTCD	24.2	68.8	14.3	23.7
No given args	8.7	20.2	3.5	6.0
No given argSTCD	8.4	26.6	3.1	5.5
Table 2: P(precision), R(ecall), and  $\mathrm{F}_1$  on RAMS development and test data. TCD designates the use of ontology-aware type-constrained decoding.
5 RAMS Experiments and Results
In the following experiments, for each event the model is given the (gold) trigger span and the (gold) spans of the arguments. The model finds for each role the best argument(s) to fill it. Predictions are returned as trigger-role-argument triples.
We use feature-based BERT-base (Devlin et al., 2018)—mixing layers 9 through 12—by splitting the documents into segments of size 512 subtokens and encoding each segment separately. $^{14}$
We perform preliminary sweeps across hyperparameter values, which are then fixed while we perform a more exhaustive sweep across scoring features. We also compare argmax decoding with greedy decoding during training. The best model is selected based on  $\mathrm{F}_1$  on the development set, and ablations are reported in Table 3. Our final model uses greedy decoding,  $s_{A,R}$ , and  $s_l$  and omits  $s_{E,R}$  and  $s_c$  (see Equation 2). More details can be found in Appendix B.
The results using our model with greedy decoding and TCD are reported in Table 2. We also report performance of the following baselines: 1) choosing for each link the most common role (place), 2) using the same fixed trigger representation across examples, and 3) using the full context window as the trigger. Additionally, we experiment with two other data conditions: 1) linking the correct argument(s) from among a set of distractor candidate arguments provided by a constituency parser (Kitaev and Klein, 2018), $^{15}$  and 2) finding the correct argument(s) from among all possible spans up to a fixed length.
<sup>13</sup>We use ELMo (Peters et al., 2018) in these experiments. He et al. (2018) achieve  $85.5\mathrm{F}_1$  with gold predicates and  $82.9\mathrm{F}_1$  without gold predicates, and Ouchi et al. (2018) achieve  $86.2\mathrm{F}_1$  with gold predicates.
$^{14}0.2\%$  of the training documents span multiple segments. We take as the distractor arguments all (potentially overlapping) NPs predicted by the parser. On average, this yields 44 distractors per training document.
8062
Model	Greedy	TCD
Our model	69.9	75.1
- distance score	69.0	74.3
- sl(a, āe,r)	54.9	58.4
- sA,R(a,r)	68.6	73.8
+ sE,R(e,r)	69.5	74.4
+ sc(e,a)	65.9	70.6
w/ argmax decoding	69.9	75.1
BERT 6–9	69.6	75.3
ELMo	68.5	75.2
For the distractor experiment, we use the same hyperparameters as for the main experiment. When not given gold argument spans, we consider all spans up to 5 tokens long and change only the hyperparameters that would prune less aggressively. We hypothesize that the low performance in this setting is due to the sparsity of annotated spans compared to the set of all enumerated spans. In contrast, datasets such as CoNLL 2012 are more densely annotated, so the training signal is not as affected when the model must determine argument spans in addition to linking them.
Finally, we examine the effect of TCD to see whether the model effectively uses gold event types if they are given. TCD filters out illegal predictions, boosting precision. Recall is still affected by this decoding strategy because the model may be more confident in the wrong argument for a given role, thus filtering out the less confident, correct one. Nevertheless, using gold types at test time generally leads to gains in performance.
Ablations Ablation studies on development data for components of the link score as well as the contextualized encoder and decoding strategy are shown in Table 3. Type-constrained decoding based on knowledge of gold event types improves  $\mathrm{F}_1$  in all cases because it removes predictions that are invalid with respect to the ontology.
The most important link score component is the score between a combined event-role and a candidate argument. This result follows intuitions that  $s_l$  is the primary component of the link score since it directly captures the compatibility of the explicit argument and the implicit argument represented by the event-role pair.
Table 3:  $\mathrm{F}_{1}$  on RAMS dev data when link score components are separately included/excluded (Equation 2) or other contextualized encoders are used in the best performing model.  $\mathrm{{TCD}} =$  type-constrained decoding.
Dist.	# Gold	# Predict	P	R	F1
-2	79 (26)	69 (21)	81.2	70.9	75.7
-1	164 (33)	151 (27)	76.8	70.7	73.7
0	1,811 (61)	1,688 (51)	77.7	72.4	75.0
1	87 (24)	83 (22)	78.3	74.7	76.5
2	47 (18)	39 (14)	87.2	72.3	79.1
Total	2,189 (62)	2,030 (52)	78.0	72.3	75.1
Table 4: Performance breakdown by distance (number of sentences) between argument and event trigger for our model using TCD over the development data. Negative distances indicate that the argument occurs before the trigger. # Gold and # Predict list the number of arguments (and unique roles) at that distance.
We also experiment with both ELMo (Peters et al., 2018) and BERT layers 6-9, which were found to have the highest mixture weights for SRL by Tenney et al. (2019a). We found that BERT generally improves over ELMo and layers 9-12 often perform better than layers 6-9.
Argument-Trigger Distance One of the differentiating components of RAMS compared to SRL datasets is its non-local annotation of arguments. At the same time, RAMS uses naturally occurring text so arguments are still heavily distributed within the same sentence as the trigger (Figure 5). This setting allows us to ask whether our model accurately finds arguments outside of the sentence containing the trigger despite the non-uniform distribution. In Table 4, we report  $\mathrm{F}_1$  based on distance on the development set and find that performance on distant arguments is comparable to performance on local arguments, demonstrating the model's ability to handle non-local arguments.
Role Embeddings and Confusion We present in Figure 4 the cosine similarities between the learned 50-dimensional role embeddings in our model and also the errors made by the model under argmax decoding on the dev set. Some roles are highly correlated. For example, origin and destination have the most similar embeddings, possibly because they co-occur frequently and have the same entity type. Conversely, negatively correlated roles have different entity types or occur in different events, such as communicator compared to destination and artifact. We also observe that incorrect predictions are made more often between highly correlated roles and err
<sup>16</sup>Analysis of the confusion matrix with type-constrained decoding is less meaningful because the constraints, which rely on gold event types, filter out major classes of errors.
8063
Figure 4: Embedding similarity (top) and row-normalized confusion (bottom) between roles for the 15 most frequent roles with our model. The full figures are included in Appendix C. Best viewed in color.
on the side of the more frequent role, as most errors occur below the diagonal.
Examples We present predictions from the development set which demonstrate some phenomena of interest. These are made without TCD, illustrating the model's predictions without knowledge of gold event types.
In Table 5, the first example demonstrates the model's ability to link a non-local argument which occurs in the sentence before the trigger. Greedy decoding helps the model find multiple arguments satisfying the same participant role, which also appear on either side of the trigger. In the second example, the model correctly predicts the driverpassenger, one of the rarer roles in RAMS (17 instances in the training set), consistent with the gold AccidentCrash event type.
In Table 6, the model fills roles corresponding to both the Death and the gold JudicialConsequences event types, thereby mixing roles from different event types. The predictions are plausible when interpreted in context and would be more accurate under TCD.
The EU's leaders in Brussels are expected to play hardball in negotiating Britain's exit, to send a message to other states that might be contemplating a similar move. "Informal meeting of EU 27 next week without PM in the room to decide common negotiating position vs UK on exit negotiations" —Faisal Islam.
SPEAKER: I'm Mary Ann Mendoza, the mother of Sergeant Brandon Mendoza, who was killed in a violent  
DRIVERPASSenger  
head-on collision in Mesa.  
PLACE
Table 5: Two examples of correct predictions on the development set.
"Many people are saying that the Iranians killed the scientist who helped the US because of Hillary Clinton's hacked emails." —8 August, Twitter. Shahran Amiri, the VICTIM, DEFENDANT nuclear scientist executed in Iran last week, ...
"Many people are saying that the Iranians killed the scientist who helped the US because of Hillary Clinton's hacked emails." —8 August, Twitter. Shahran Amiri, the nuclear scientist executed in Iran last week, ...
Table 6: A partially correct prediction (top) and its corresponding gold annotations (bottom).
5.2 AIDA Phase 1
We also investigate how well RAMS serves as pre-training data for AIDA-1. A model using the hyperparameters of our best-performing RAMS model and trained on just English AIDA-1 Practice data achieves  $19.1\mathrm{F}_1$  on the English AIDA-1 Eval data under greedy decoding and  $18.2\mathrm{F}_1$  with TCD. When our best-performing RAMS model is fine-tuned to the AIDA task by further training on the AIDA-1 data, performance is improved to  $24.4\mathrm{F}_1$  under greedy decoding and  $24.8\mathrm{F}_1$  with TCD. The crowdsourced annotations in RAMS are therefore of sufficient quality to serve as augmentation to LDC's AIDA-1. Experimental details are available in Appendix D.
6 Other Datasets
6.1 Beyond NomBank
The Beyond NomBank (BNB) dataset collected by Gerber and Chai (2010) and refined by Gerber and Chai (2012) contains nominal predicates (event triggers) and multi-sentence arguments, both of which are properties shared with RAMS.
To accommodate our formulation of the argu
8064
Field	Baseline*	Our Model
Victim Name	9.3 (54.1)	62.2 (69.6)
Shooter Name	4.7 (24.1)	53.1 (57.8)
Location	12.2 (18.9)	34.9 (63.3)
Time	68.1 (69.3)	62.9 (69.4)
Weapon	1.1 (17.9)	32.5 (49.6)
Table 7: Strict (and approximate) match  $\mathrm{F}_1$  on GVDB. Due to the different data splits and evaluation conditions, we are not directly comparable to the baseline (Pavlick et al., 2016), provided only for reference.
ment linking task, we modify the BNB data in two ways: 1) we merge "split" arguments, which in all but one case are already contiguous spans; and 2) we reduce each cluster of acceptable argument fillers to a set containing only the argument closest to the trigger. We also make modifications to the data splits for purposes of evaluation. Gerber and Chai (2012) suggest evaluation be done using cross-validation on shuffled data, but this may cause document information to leak between the train and evaluation folds. To prevent such leakage and to have a development set for hyperparameter tuning, we separate the data into train, dev, and test splits with no document overlap. Additional data processing details and hyperparameters are given in Appendix E. When given gold triggers and argument spans, our model achieves  $75.4\mathrm{F}_1$  on dev data and  $76.6\mathrm{F}_1$  on test data.
6.2 Gun Violence Database
The Gun Violence Database (GVDB) (Pavlick et al., 2016) is a collection of news articles from the early 2000s to 2016 with annotations specifically related to a gun violence event. We split the corpus chronologically into a training set of 5,056 articles, a development set of 400, and a test set of 500. We use this dataset to perform a MUC-style information extraction task (Sundheim, 1992). While GVDB's schema permits any number of shooters or victims, we simply predict the first mention of each type. Pavlick et al. (2016) perform evaluation in two settings: a strict match is awarded if the predicted string matches the gold string exactly, while an approximate match is awarded if either string contains the other.
Assuming each document contains a single gun violence event triggered by the full document, our goal is to predict the value (argument) for each slot (role) for the event. As each slot is filled by exactly one value, we use argmax decoding.
While the baseline experiments of Pavlick et al. (2016) made sentence-level predictions focusing on five attributes, we make document-level predictions and consider the larger set of attributes. Table 7 shows our model's performance on the shared subset of attributes, but the numerical values are not directly comparable because the prior work makes predictions on the full dataset and also combines some roles. Our results show that our model is suitable for information extraction tasks like slot filling. Appendix F contains information on hyperparameters and performance on the full set of roles. To our knowledge, our results are a substantial improvement over prior attempts to predict attributes of gun violence event reports, and we make our models available in the hopes of assisting social scientists in their corpus studies.
We introduced a novel model for document-level argument linking. Because of the small amount of existing data for the task, to support training our neural framework we constructed the RAMS dataset consisting of 9,124 events covering 139 event types. Our model outperforms strong baselines on RAMS, and we also illustrated its applicability to a variety of related datasets. We hope that RAMS will stimulate further work on multisentence argument linking.
We thank Craig Harman for his help in developing the annotation interface. We also thank Tongfei Chen, Yunmo Chen, members of JHU CLSP, and the anonymous reviewers for their helpful discussions and feedback. This work was supported in part by DARPA AIDA (FA8750-18-2-0015) and IARPA BETTER (#2019-19051600005). The views and conclusions contained in this work are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, or endorsements of DARPA, ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998. The Berkeley FrameNet project. In COLING
8065
1998 Volume 1: The 17th International Conference on Computational Linguistics.
Aljoscha Burchardt, Katrin Erk, Anette Frank, Andrea Kowalski, and Sebastian Pado. 2006. SALTO - a versatile multi-level annotation tool. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC'06), Genoa, Italy. European Language Resources Association (ELRA).
Yubo Chen, Liheng Xu, Kang Liu, Daojian Zeng, and Jun Zhao. 2015. Event extraction via dynamic multi-pooling convolutional neural networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 167-176, Beijing, China. Association for Computational Linguistics.
Pengxiang Cheng and Katrin Erk. 2018. Implicit argument prediction with event knowledge. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 831-840, New Orleans, Louisiana. Association for Computational Linguistics.
Pengxiang Cheng and Katrin Erk. 2019. Implicit argument prediction as reading comprehension. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6284-6291.
Dipanjan Das, Nathan Schneider, Desai Chen, and Noah A. Smith. 2010. Probabilistic frame-semantic parsing. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 948-956, Los Angeles, California. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.
Parvin Sadat Feizabadi and Sebastian Padó. 2014. Crowdsourcing annotation of non-local semantic roles. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, volume 2: Short Papers, pages 226-230, Gothenburg, Sweden. Association for Computational Linguistics.
Parvin Sadat Feizabadi and Sebastian Padó. 2015. Combining seemingly incompatible corpora for implicit semantic role labeling. In Proceedings of the Fourth Joint Conference on Lexical and Computational Semantics, pages 40-50, Denver, Colorado. Association for Computational Linguistics.
Charles J Fillmore. 1986. Pragmatically controlled zero anaphora. In Annual Meeting of the Berkeley Linguistics Society, volume 12, pages 95-107.
Charles J. Fillmore, Collin F. Baker, and Hiroaki Sato. 2002. The FrameNet database and software tools. In Proceedings of the Third International Conference on Language Resources and Evaluation (LREC'02), Las Palmas, Canary Islands - Spain. European Language Resources Association (ELRA).
Matthew Gerber and Joyce Chai. 2010. Beyond NompBank: A study of implicit arguments for nominal predicates. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1583-1592, Uppsala, Sweden. Association for Computational Linguistics.
Matthew Gerber and Joyce Y. Chai. 2012. Semantic role labeling of implicit arguments for nominal predicates. Computational Linguistics, 38(4):755-798.
Luheng He, Kenton Lee, Omer Levy, and Luke Zettle-moyer. 2018. Jointly predicting predicates and arguments in neural semantic role labeling. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 364-369, Melbourne, Australia. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Nikita Kitaev and Dan Klein. 2018. Constituency parsing with a self-attentive encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2676–2686, Melbourne, Australia. Association for Computational Linguistics.
Kevin Knight, Bianca Badarau, Laura Baranescu, Claire Bonial, Madalina Bardocz, Kira Griffith, Ulf Hermjakob, Daniel Marcu, Martha Palmer, Tim O'Gorman, and Nathan Schneider. 2020. Abstract Meaning Representation (AMR) annotation release 3.0 LDC2020T02. Linguistic Data Consortium, Philadelphia, PA.
Kenton Lee, Luheng He, Mike Lewis, and Luke Zettle-moyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188-197, Copenhagen, Denmark. Association for Computational Linguistics.
Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 687-692, New Orleans, Louisiana. Association for Computational Linguistics.
Thien Huu Nguyen, Kyunghyun Cho, and Ralph Grishman. 2016. Joint event extraction via recurrent
8066
neural networks. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 300-309, San Diego, California. Association for Computational Linguistics.
Tim O'Gorman, Michael Regan, Kira Griffith, Ulf Hermjakob, Kevin Knight, and Martha Palmer. 2018. AMR beyond the sentence: the multi-sentence AMR corpus. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3693-3702, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Timothy J O'Gorman. 2019. Bringing Together Computational and Linguistic Models of Implicit Role Interpretation. PhD dissertation, University of Colorado at Boulder.
Hiroki Ouchi, Hiroyuki Shindo, and Yuji Matsumoto. 2018. A span selection model for semantic role labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1630-1642, Brussels, Belgium. Association for Computational Linguistics.
Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The proposition bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1):71-106.
Ellie Pavlick, Heng Ji, Xiaoman Pan, and Chris Callison-Burch. 2016. The gun violence database: A new task and data set for NLP. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1018-1024, Austin, Texas. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards robust linguistic analysis using OntoNotes. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pages 143-152, Sofia, Bulgaria. Association for Computational Linguistics.