SlowGuess/ACL_2020_OCR huggingface dataset:
The entire GVDB corpus consists of 7,366 articles. We exclude articles that do not have a reliable publication date or lack annotated spans for the roles we are interested in. Additionally, a buffer of 100 articles spanning roughly one week between the dev and test set is discarded, limiting the possibility of events occurring in both the development and test sets. We also filter out spans whose start and end boundaries are in different sentences, as these are unlikely to be well-formed argument spans. For evaluation, a slot's value is marked as correct under the strict setting if any of the predictions for that slot match the string of the correct answer exactly, while an approximate match is awarded if either a prediction contains the correct answer or if the correct answer contains the predicted string. The approximate setting is necessary due to inconsistent annotations (e.g., omitting first or last names).
We experiment with the feature-based version of BERT-base and with ELMo as our contextualized encoder. Table 11 lists the numerical hyper
parameters for this model. Since there is only one event per document and no explicit trigger,  $e$  is represented by a span embedding of the full document. We use the top four layers (9-12) of BERT-base cased (all three layers for ELMo) with a lexical dropout of 0.5. Everywhere else, we apply a dropout of 0.4. We train with the Adam optimizer (Kingma and Ba, 2015) and use patience-based early stopping. Our best checkpoint was after 8 epochs (roughly 9 hours on a single NVIDIA 1080Ti). Even though the official evaluation is string based, we used a span-based micro  $\mathrm{F}_1$  metric for early stopping.
For this model,  $\phi_{l}$  corresponds to a learned (bucketed) positional embedding of the argument span (i.e., distance from the start of the document). In computing the coarse score, we omit  $\phi_c$ . When computing Equation 2, we omit  $s_{A,R}$  but keep all other terms in Equation 2. We adopt the character convolution of 50 8-dimensional filters of window sizes 3, 4, and 5 (Lee et al., 2017).
With the same hyperparameters and feature choices, we perform an identical evaluation using ELMo instead of BERT. As the original documents are not tokenized, we use SpaCy 2.1.4 for finding sentence boundaries and tokenization. The complete list of annotated fields are VICTIM (name, age, race), SHOOTER (name, age, race), LOCATION (specific location<sup>19</sup> or city), TIME (time of day or clock time) and WEAPON (weapon type, number of shots fired). While Pavlick et al. (2016) only make predictions for VICTIM_NAME, SHOOTER_NAME, LOCATION.(CITY|LOCATION),
TIME.(TIME|CLOCK), and WEAPON.WEAPON, we perform predictions over all annotated span-based fields. The full results for both BERT and ELMo are reported in Table 12 and Table 13, respectively. BERT generally improves over ELMo across the board, but not by a sizeable margin. Despite the inability to directly compare, we nonetheless present a stronger and more comprehensive baseline for future work with GVDB.
19For example, a park or a laundromat.
8075
Figure 13: Full version of Figure 4, showing cosine similarity between role embeddings. Best viewed in color.
Field	Strict	Partial
Baseline	F1	P	UsR	F1	Baseline	F1	P	UsR	F1
P	R	P	R
VICTIM	Name	10.2	8.5	9.3	61.2	63.3	62.2	59.5	49.6	54.1	68.4	70.9	69.6
Age	-	-	-	19.4	24.2	21.5	-	-	-	67.3	84.1	74.8
Race	-	-	-	75.5	74.1	74.8	-	-	-	75.5	74.1	74.8
SHOOTER	Name	5.8	3.9	4.7	55.3	51.1	53.1	30.2	20.1	24.1	60.2	55.6	57.8
Age	-	-	-	34.1	32.6	33.3	-	-	-	69.0	65.9	67.4
Race	-	-	-	72.7	55.2	62.7	-	-	-	81.8	62.1	70.6
LOCATION	CityLocation	19.9	8.8	12.2	67.436.1	66.233.8	66.834.9	30.8	13.6	18.9	72.265.4	70.961.2	71.563.3
TIME	TimeClock	69.3	66.9	68.1	57.244.0	69.747.6	62.945.7	70.5	68.1	69.3	63.284.0	76.990.8	69.487.2
WEAPON	WeaponNum Shots	2.1-	0.7-	1.1-	33.340.6	31.711.2	32.517.6	36.8-	11.8-	17.9-	50.962.5	48.317.2	49.627.0
Table 12: P(precision), R(ecall), and  $F_1$  on event-based slot filling (GVDB) using BERT as the document encoder. Due to the different data splits and evaluation conditions, the results are not directly comparable to the baseline (Pavlick et al., 2016), which is provided only for reference. Fields that were aggregated in the baseline are predicted separately in our model.  $-$  indicates result is not reported in the baseline.
8076
Figure 14: Full version of Figure 4, showing row-normalized confusion between roles. Note that roles not predicted at all would result in empty rows and so are omitted from the table.
Field	Strict	Partial
Baseline	F1	P	UsR	F1	Baseline	F1	P	UsR	F1
P	R	P	R
VICTIM	Name	10.2	8.5	9.3	56.2	56.8	56.5	59.5	49.6	54.1	62.7	63.3	63.0
Age	-	-	-	29.5	33.9	31.6	-	-	-	64.4	74.0	68.9
Race	-	-	-	73.2	75.9	74.5	-	-	-	75.0	77.8	76.4
SHOOTER	Name	5.8	3.9	4.7	53.7	60.2	56.7	30.2	20.1	24.1	56.4	63.2	59.6
Age	-	-	-	27.3	31.8	29.4	-	-	-	53.2	62.1	57.3
Race	-	-	-	55.9	65.5	60.3	-	-	-	58.8	69.0	63.5
LOCATION	CityLocation	19.9	8.8	12.2	59.136.6	61.134.7	60.135.6	30.8	13.6	18.9	64.159.1	66.256.0	65.157.5
TIME	TimeClock	69.3	66.9	68.1	57.744.6	64.745.8	61.045.2	70.5	68.1	69.3	64.583.5	72.485.6	68.284.5
WEAPON	Weapon	2.1	0.7	1.1	32.7	26.7	29.4	36.8	11.8	17.9	44.9	36.7	40.4
Num Shots	-	-	-	23.3	18.1	20.4	-	-	-	42.2	32.8	36.9
Table 13: P(decision), R(ecall), and  $\mathrm{F}_{1}$  on event-based slot filling (GVDB) using ELMo at the sentence level. On average, the performance is outperformed by BERT.
8077