SlowGuess/ACL_2020_OCR huggingface dataset:
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Michael Roth and Anette Frank. 2013. Automatically identifying implicit arguments to improve argument linking and coherence modeling. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity, pages 306-316, Atlanta, Georgia, USA. Association for Computational Linguistics.
Josef Ruppenhofer, Caroline Sporleder, Roser Morante, Collin Baker, and Martha Palmer. 2010. SemEval-2010 task 10: Linking events and their participants in discourse. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 45-50, Uppsala, Sweden. Association for Computational Linguistics.
Niko Schenk and Christian Chiarcos. 2016. Unsupervised learning of prototypical fillers for implicit semantic role labeling. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1473-1479, San Diego, California. Association for Computational Linguistics.
Carina Silberer and Anette Frank. 2012. Casting implicit role linking as an anaphora resolution task. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 1–10, Montreal, Canada. Association for Computational Linguistics.
Linfeng Song, Yue Zhang, Zhiguo Wang, and Daniel Gildea. 2018. N-ary relation extraction using graph-state LSTM. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2226–2235, Brussels, Belgium. Association for Computational Linguistics.
Beth M. Sundheim. 1992. Overview of the fourth message understanding evaluation and conference. In FOURTHMESSAGE UNDERSTANDING CONFERENCE (MUC-4), Proceedings of a Conference Held in McLean, Virginia, June 16-18, 1992.
Swabha Swayamdipta, Sam Thomson, Kenton Lee, Luke Zettlemoyer, Chris Dyer, and Noah A. Smith. 2018. Syntactic scaffolds for semantic structures. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3772-3782, Brussels, Belgium. Association for Computational Linguistics.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019a. BERT rediscovers the classical NLP pipeline. In
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593-4601, Florence, Italy. Association for Computational Linguistics.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. 2019b. What do you learn from context? Probing for sentence structure in contextualized word representations. In International Conference on Learning Representations.
Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, Mohammed El-Bachouti, Robert Belvin, and Ann Houston. 2013. OntoNotes release 5.0 LDC2013T19. Linguistic Data Consortium, Philadelphia, PA.
Travis Wolfe, Mark Dredze, and Benjamin Van Durme. 2015. Predicate argument alignment using a global coherence model. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 11-20, Denver, Colorado. Association for Computational Linguistics.
8068
A RAMS Data
A.1 Collection
On Reddit, users make submissions containing links to news articles, images, videos, or other kinds of documents, and other users may then vote or comment on the submitted content. We collected news articles matching the following criteria: 1) Posted to the  $r$ /politics sub-forum between January and October 2016; 2) Resulted in threads with at least 25 comments; and 3) Contained at least one mention of the string "Russia". The resulting subset of articles tended to describe geopolitical events and relations like the ones in the AIDA ontology. In order to filter out low-quality, fake, or disreputable news articles, we treat the number of comments in the discussion as a signal of information content. Our approach of gathering user-submitted and curated content through Reddit is similar to those used for creating large datasets for language model pre-training (Radford et al., 2019). Documents were split into sentences using NLTK 3.4.3, and sentences were split into tokens using SpaCy 2.1.4.
To assess whether a lexical unit (LU) evoked an event with positive factuality, the vetting task contained an event definition and several candidate sentences, each with a highlighted LU. Annotators were asked to judge how well each highlighted LU, in the context of its sentence, matched the provided event definition. In the same task, they were also asked to assess the factuality of the sentence. Annotation instructions and examples are shown in Figure 9 and Figure 10.
Figure 5: Distances between triggers and arguments in RAMS and proportion of arguments at that distance (counts are shown above each bar). Negative distances indicate that the argument occurs before the trigger.
Each argument selection task contained five
tokenized sentences, a contiguous set of tokens marking the trigger, a definition of the event type, and a list of roles and their associated definitions. For each role, annotators were asked whether a corresponding argument was present in the 5-sentence window, and if so, to highlight the argument span that was closest to the event trigger, as there could be multiple. In cases near the beginning or end of a document, annotators were shown up to two sentences before or after the sentence containing the trigger. Annotators were allowed to highlight any set of (within-sentence) contiguous tokens within the 5-sentence window aside from the trigger tokens. The distribution of distances between triggers and arguments is shown in Figure 5. Annotation instructions and an example are shown in Figure 11 and Figure 12.
A.3 Agreement
We additionally compute the frequency with which annotators agreed a given role was or was not present in the context window. To measure the frequency with which annotators agree whether a given role is present, we treat the majority annotation as the gold standard. Then, we calculated the precision, recall, and  $\mathrm{F}_1$  of the annotations. Across the set of redundantly annotated tasks, there were 83 false negatives, 60 false positives, and 892 true positives, giving a precision of 93.7, recall of 91.5, and an  $\mathrm{F}_1$  of 92.6.
Threshold	Conjunctive	Disjunctive	Start	End
0	55.3	78.0	59.8	73.5
1	69.9	80.3	74.9	75.3
2	73.9	82.0	78.2	77.8
3	76.4	83.6	80.9	79.1
4	78.8	84.3	82.7	80.4
Table 8: Pairwise span boundary inter-annotator agreement statistics for various span difference thresholds.
We consider a wider range of span difference thresholds, where span difference is calculated by using the absolute difference of the (start, end) token indices from each pair. These are presented in Table 8. In conjunctive agreement, both  $|start_1 - start_2|$  and  $|end_1 - end_2|$  must be less than the given threshold; therefore, conjunctive agreement at threshold 0 is the percent of pairs that exactly agree (55.3%). Disjunctive agreement is less strict, requiring that either the absolute difference of start offsets or end offsets must be less than the threshold. Start and end agreement is deter
8069
Figure 6: Comparison of frequency (top) and amount of dataset covered (bottom) of event types sorted by decreasing frequency. RAMS has more annotations for a more diverse set of event types than do AIDA Phase 1 and Beyond NomBank.
mined by considering whether the absolute difference of the pair's start or end offsets (respectively) is within the given threshold.
A.4 Event and Role Type Coverage
Event type and role type coverage are shown in Figure 6 and Figure 7. Figure 6 illustrates that RAMS contains more annotations for a larger set of event types than does AIDA-1. In addition, the distribution of annotations in RAMS is less skewed (more entropic) than in AIDA-1, in that in order to cover a given percentage of the dataset, more event types must be considered in RAMS than in AIDA-1. Figure 7 shows a similar pattern for role type coverage.
Figure 8 shows role coverage per event type, a measure of how much of each event type's role set is annotated on average. Role coverage per event type is calculated as the average number of filled roles per instance of the event type divided by the number of roles specified for that event type by the ontology. For the RAMS training set, the  $25^{\text{th}}$  percentile is  $55.6\%$ , the  $50^{\text{th}}$  percentile is  $61.9\%$ , and the  $75^{\text{th}}$  percentile is  $68.6\%$  coverage.
Figure 7: Comparison of frequency (top) and amount of dataset covered (bottom) of roles sorted by decreasing frequency. RAMS has more annotations for a more diverse set of role types than the AIDA Phase 1 data.
B RAMS Hyperparameters
Table 9 lists the numerical hyperparameters shared by all models discussed in this paper. Models may ignore some link score components if they were found to be unhelpful during our sweep of Equation 2 and Equation 3. For our model, we learn a linear combination of the top layers (9, 10, 11, 12) of BERT-base cased, while we use the middle layers (6, 7, 8, 9) for the 6-9 ablation. For ELMo, we use all three layers and encode each sentence separately. We apply a lexical dropout of 0.5 to these embeddings.
Figure 8: Number of event types for which a given percentage of roles are filled in RAMS train set.
8070
Below, you will be presented with a description of an event and several news article sentences. Each sentence contains a piece of highlighted text. In short, we are interested in whether the highlighted text "matches" the provided event description.
For each sentence, you will answer up to three questions. First, we establish whether the text contains any kind of event:
1. Does the highlighted piece of text refer to an event?
Important:
- Most verbs refer to events, but nouns can refer to events too! For example, "destruction" or "robbery".
- It's fine if just one or two words are highlighted, as long as those words refer to an event.
- For example, "the crash was relatively minor".
- If the sentence contains an event but it is not highlighted, answer "no" to this question.
Then, if the highlighted text does refer to an event, we ask:
2. How well does the highlighted event match the provided event description?
The event can match even if the event did not occur! Imagine we have the event description: "an object is involved in a collision".
In "the crash was narrowly avoided", the event matches the description even though the crash didn't occur!
- We ask a separate question to determine whether the event happened or not.
- You may have to distinguish between several senses of a word.
In "the stock market crash of 1987", the "crash" is not a "collision" event.
- If the sentence contains a matching event, but it is not highlighted, answer "no" to this question.
Finally, we ask:
3. How strongly does the sentence imply that the highlighted event actually happened?
- Pay attention to words like "definitely", "undoubtedly", "unlikely", "maybe", etc.
- In the absence of any obvious clues, you should lean towards "yes".
- Only use the information in the text! To the extent possible, don't use any background knowledge.
For example, in "Ben crashed into a unicorn", the text implies the event occurred, even though we know it couldn't have happened.
If the sentence implies the event happened or is presently happening, lean towards "yes". If the event will happen in the future, lean towards "no".
Event Name: Collision
Event Description: An object collides violently with an obstacle or another object
Sentence: The car crash was narrowly avoided.
Does the highlighted piece of text refer to an event?
Figure 9: Annotation instructions for determining whether a lexical unit (in context) evokes an event type.
How well does the highlighted event match the provided event description?
How strongly does the sentence imply that the highlighted event actually happened?
Figure 10: Annotation interface for determining whether a lexical unit (in context) evokes an event type.
8071
This task is helping us build systems to better understand documents, thank you for your help.
You will be presented with up to five sentences that have been extracted from a larger document. In one of those sentences an event trigger has been highlighted: some word or phrase that refers to an event. We need your help to highlight the arguments of that event, somewhere in the provided text. "Arguments" are just the entities that play a certain role in an event; for example, in "John ate the sandwich" we could say that "John" is the "Consumer" and "the sandwich" is the "ConsumedObject".
The arguments are defined in two ways:
1. Each argument has a role name, which sometimes can be informative such as Transporter, and other times may not be very helpful, such as GPE
2. We provide a gloss for the event, where we have written a sentence based on the event definition, with the arguments initially filled in with words like someone or someplace.
Additionally, you will be provided with the name and definition of the event. If you find it hard to interpret a particular argument, we suggest re-reading the event name and definition and seeing how that argument fits in with the event.
As you highlight argument spans, the gloss will update based on your selection. The gloss is meant to be a helpful tool: it can happen that as you select arguments it will not always result in a perfectly grammatical gloss. That situation is OK, just do your best.
We would like you to select arguments even if the event did not happen! For example, in "John might crash his car tomorrow", no "Crash" event occurred yet, but we'd still like you to highlight arguments.
Finally, since there may be several text spans that refer to the same entity, we ask that you select the one that is closest to the highlighted trigger word. For example, if we are trying to find the "Consumer" in the sentence: "John and Mary like candy. They ate some today", we ask that you select "they", rather than "John and Mary", since "they" is closer to the trigger.
Sometimes not all of the arguments will be present in the text; in that case, we ask you explicitly check the box that a given argument can not be found. The following is an example of a completed task where the "Place" argument is not mentioned:
Figure 11: Annotation instructions for selecting arguments for an event.
Figure 12: Annotation interface for selecting arguments for an event.
8072
Hyperparameter	Value
Embeddings	role size	50
feature (φl) size	20
size	200
LSTM	layers	3
dropout	0.4
argument (FA)	size	150
layers	2
event-role (FE,R)	size	150
layers	2
F̅a (Eqn. 1)	layers	2
arg-role (FA,R)	size	150
layers	2
Fl	size	150
layers	2
distance FFNN	size	150
layers	2
Pruning	k	10
Memory Limits	training doc size	1000
batch size	1
learning rate	0.001
Training	decay	0.999
patience	100 steps
Table 9: Hyperparameters of the model trained on RAMS. Sizes of learned weights that are omitted from the table can be determined from these hyperparameters. As the argument spans are given to the model in our experiments, we skip the first pass of pruning. We do not clip gradients.
In our best model, we use learned bucketed distance embeddings (Lee et al., 2017). These embeddings are scored as part of  $\phi_c$  in computing  $s_c(e,a)$  in Equation 2 and are also scored as a part of  $\phi_l$  in  $s_l$  (Equation 3). Since span boundaries are given in our primary experiments, we do not include a score  $s_A$  or  $s_E$  in  $s_c$ . Our best model uses both  $s_{A,R}$  and  $s_l(a,\tilde{a}_{e,r})$  in Equation 2. These features were chosen as the result of a sweep over possible features, with other ablations reported in Table 3.
We adopt the span embedding approach by Lee et al. (2017), which uses character convolutions (50 8-dimensional filters of sizes 3, 4, and 5) and 300-dimensional GloVe embeddings. The default dropout applied to all connections is 0.2. We optimize using Adam (Kingma and Ba, 2015) with patience-based early stopping, resulting in the best checkpoint after 19 epochs (9 hours on an NVIDIA 1080Ti), using  $\mathrm{F}_1$  as the evaluation metric.
Hyperparameters for the condition with distractor candidate arguments are the same as those in Table 9. For the condition with no given argument spans, we consider all intrasentential spans
up to 5 tokens in length. We include the score of each candidate argument span when pruning to encourage the model to keep correct spans. We modify hyperparameters in Table 9 to prune less aggressively, setting  $k = 100$  and  $\lambda_{A} = 1.0$  (defined in §4.1).
C Full Role Confusion and Similarity Matrices
Figure 13 shows the similarity between all 65 role embeddings, while Figure 14 visualizes all the errors made by the model on the development set. These are expansions of the per-role results from §5.1.
Since argument linking is not a one-to-one labeling problem, we need to perform a modified procedure for visualizing a confusion matrix. For example, an argument span may take on multiple roles for the same event. To compute the errors, we first align the correct prediction(s) and subsequently compute the errors for the remaining gold and predicted label(s). For example, if the correct set of roles is {destination, origin} and the model predicts {origin, place}, then we only mark place as an error for destination.
D AIDA Phase 1
D.1 Data Processing
We filter and process the AIDA-1 Practice and Eval data in the following way. Because annotations are available for only a subset of the documents in AIDA-1, we consider only the documents that have textual event triggers. We then take from this set only the English documents, which, due to noisy language ID in the original annotations, were selected by manual inspection of the first 5 sentences of each document by one of the authors of this work.
In addition, the argument spans in each example are only those that participate in events. In other words, arguments of relations (that are not also arguments of events) are not included. Additionally, a document may contain multiple events, unlike in RAMS.
The training and development set come from AIDA-1 Practice, and the test set comes from AIDA-1 Eval. As the AIDA-1 Eval documents are about different topics than the Practice documents are, we emulate the mismatch in topic distribution by using a development set that is about a different
8073
Strategy	Dev. F1	P	R	F1
No pre-training	25.0	36.6	12.9	19.1
No pre-trainingTCD	27.1	53.5	11.0	18.2
RAMS pre-training	34.1	43.9	16.9	24.4
RAMS pre-trainingTCD	34.2	62.5	15.4	24.8
Table 10: P(precision), R(ecall), and  $\mathrm{F}_1$  on AIDA-1 English development and test data. TCD designates the use of ontology-aware type-constrained decoding.
topic than the training set is. We use Practice topics R103 and R107 for training and R105 for development because R105 is the smallest of the three practice topics both by number of documents and by number of annotations. The test set consists of all 3 topics (E101, E102, E103) from the (unsequested) Eval set. After the filtering process described above, we obtain a training set of 46 documents, a development set of 17 documents, and a test set of 69 documents. There are 389 events in the training set, and the training documents have an average length of 50 sentences.
We use the same hyperparameters as the best model for RAMS, shown in Table 9.
D.3 Pre-training on RAMS
Both the models with and without pre-training on RAMS were trained on AIDA-1 for 100 epochs with an early-stopping patience of 50 epochs using the same hyperparameters as the best RAMS model. All parameters were updated during finetuning (none were frozen). The vocabulary of the pre-trained model was not expanded when trained on AIDA-1.
The models' lower performance on AIDA-1 than on RAMS may be in part explained by the presence of distractors in AIDA-1. Moving from RAMS (one trigger per example) to AIDA-1 (many triggers per example) introduces distractor "negative" links: an argument for one event might not participate in a different event in the same document. When given gold argument spans, a model learns from RAMS that every argument gets linked to the trigger, but there are many negative links in the AIDA-1 data, which the model must learn to not predict.
Full results are given in Table 10. Type-constrained decoding does not improve performance on AIDA-1 as much as it did in Table 3, possibly because the AIDA-1 data often does not
adhere to the multiplicity constraints of the ontology. For example, many attack events have more than one annotated attacker or target. Under TCD, correct predictions made in excess of what the ontology allows are deleted, hurting recall.
Interestingly, type-constrained decoding hurts performance on AIDA-1 Eval when there is no pre-training. As discussed in §5, type-constrained decoding tends to improve precision and lower recall. Despite the same behavior here,  $\mathrm{F}_1$  is nonetheless decreased.
We see similar behavior in this experiment to the RAMS experiment involving distractor candidate arguments: low performance which is reduced further when using TCD.
E BNB Data Processing and Hyperparameters
E.1 Data Processing
We use the data from Gerber and Chai (2012). We processed the data in the following way. The annotations were first aligned to text in the Penn Treebank. Because our model assumes that arguments are contiguous spans, we then manually merged all "split" arguments, which with one exception were already contiguous spans of text. For the one split argument that was not a contiguous span, we replaced it with its maximal span. We then removed special parsing tokens such as "trace" terminals from the text and realigned the spans. While BNB gives full credit as long as one argument in each argument "cluster" is found, our training objective assumes one argument per role. We therefore automatically reduced each argument cluster to a singleton set containing the argument closest to the trigger. This reformulation of the problem limits our ability to compare to prior work.
Once all the data had been processed, we created training, development, and test splits. To avoid leaking information across splits, we bucketed examples by document and randomly assigned documents to the splits so that the splits contained instances in the proportions  $80\%$  (train),  $10\%$  (dev), and  $10\%$  (test).
<sup>17</sup>http://lair.cse.msu.edu/projects/implicit Argument_annotation.zip. Information about the data and its fields is available at http://lair.cse.msu.edu/projects/implicit_annotation.html.
18The instance is a quote broken by speaker attribution, where the split argument consists of the two halves of the quote. This example appears in our training set.
8074
Hyperparameter	Value
Embeddings	role size	50
feature (φl) size	20
size	200
LSTM	layers	3
dropout	0.4
argument (FA)	size	150
layers	2
event-role (FE,R)	size	150
layers	2
Fā (Eqn. 1)	layers	2
Fl	size	150
layers	2
positional FFNN	size	150
layers	2
# buckets	10
Pruning	λA	0.8
k	45
Memory Limits	training doc size	600
span width	15
batch size	1
Training	learning rate	0.0005
decay	0.999/200 steps
patience	20
gradient clipping	10.0
Table 11: Hyperparameters of the model trained on GVDB.
F GVDB Hyperparameters and Additional Results