SlowGuess/ACL_2020_OCR huggingface dataset:
Aakanksha Naik
anaik@cs.cmu.edu
Carolyn Rosé
cprose@cs.cmu.edu
We tackle the task of building supervised event trigger identification models which can generalize better across domains. Our work leverages the adversarial domain adaptation (ADA) framework to introduce domain-invariance. ADA uses adversarial training to construct representations that are predictive for trigger identification, but not predictive of the example's domain. It requires no labeled data from the target domain, making it completely unsupervised. Experiments with two domains (English literature and news) show that ADA leads to an average F1 score improvement of 3.9 on out-of-domain data. Our best performing model (BERT-A) reaches 44-49 F1 across both domains, using no labeled target data. Preliminary experiments reveal that finetuning on  $1\%$  labeled data, followed by self-training leads to substantial improvement, reaching 51.5 and 67.2 F1 on literature and news respectively. $^{1}$
Events are a key semantic phenomenon in natural language understanding. They embody a basic function of language: the ability to report happenings. Events are a basic building block for narratives across multiple domains such as news articles, stories and scientific abstracts, and are important for many downstream tasks such as question answering (Saurí et al., 2005) and summarization (Daniel et al., 2003). Despite their utility, event extraction remains an onerous task. A major reason for this is that the notion of what counts as an "event" depends heavily on the domain and task at hand. For example, should a system which extracts events from doctor notes only focus on medical events (eg: symptoms, treatments), or also annotate lifestyle events (eg: dietary changes, ex
ercise habits) which may have bearing on the patient's illness? To circumvent this, prior work has mainly focused on annotating specific categories of events (Grishman and Sundheim, 1996; Doddington et al., 2004; Kim et al., 2008) or narratives from specific domains (Pustejovsky et al., 2003; Sims et al., 2019). This has an important implication for supervised event extractors: they do not generalize to data from a different domain or containing different event types (Keith et al., 2017). Conversely, event extractors that incorporate syntactic rule-based modules (Saurí et al., 2005; Chambers et al., 2014) tend to overgenerate, labeling most verbs and nouns as events. Achieving a balance between these extremes will help in building generalizable event extractors, a crucial problem since annotated training data may be expensive to obtain for every new domain.
Prior work has explored unsupervised (Huang et al., 2016; Yuan et al., 2018), distantly supervised (Keith et al., 2017; Chen et al., 2017; Araki and Mitamura, 2018; Zeng et al., 2018) and semi-supervised approaches (Liao and Grishman, 2010; Huang and Riloff, 2012; Ferguson et al., 2018), which largely focus on automatically generating in-domain training data. In our work, we try to leverage annotated training data from other domains. Motivated by the hypothesis that events, despite being domain/ task-specific, often occur in similar contextual patterns, we try to inject lexical domain-invariance into supervised models, improving generalization, while not overpredicting events.
Concretely, we focus on event trigger identification, which aims to identify triggers (words) that instantiate an event. For example, in "John was born in Sussex", born is a trigger, invoking a BIRTH event. To introduce domain-invariance, we adopt the adversarial domain adaptation (ADA) framework (Ganin and Lempitsky, 2015) which constructs representations that are predictive for trigger
1Our system is available at https://github.com/aakanksha19/ODETTE
7618
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7618-7624 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
identification, but not predictive of the example's domain, using adversarial training. This framework requires no labeled target domain data, making it completely unsupervised. Our experiments with two domains (English literature and news) show that ADA makes supervised models more robust on out-of-domain data, with an average F1 score improvement of 3.9, at no loss of in-domain performance. Our best performing model (BERT-A) reaches 44-49 F1 across both domains using no labeled data from the target domain. Further, preliminary experiments demonstrate that finetuning on  $1\%$  labeled data, followed by self-training leads to substantial improvement, reaching 51.5 and 67.2 F1 on literature and news respectively.
2 Approaching Open Domain Event Trigger Identification
Throughout this work, we treat the task of event trigger identification as a token-level classification task. For each token in a sequence, we predict whether it is an event trigger. To ensure that our trigger identification model can transfer across domains, we leverage the adversarial domain adaptation (ADA) framework (Ganin and Lempitsky, 2015), which has been used in several NLP tasks (Ganin et al., 2016; Li et al., 2017; Liu et al., 2017; Chen et al., 2018; Shah et al., 2018; Yu et al., 2018).
2.1 Adversarial Domain Adaptation
Figure 1 gives an overview of the ADA framework for event trigger identification. It consists of three components: i) representation learner  $(R)$  ii) event classifier  $(E)$  and iii) domain predictor  $(D)$ . The representation learner generates token-level representations, while the event classifier and domain predictor use these representations to identify event triggers and predict the domain to which the sequence belongs. The key idea is to train the representation learner to generate representations which are predictive for trigger identification but not predictive for domain prediction, making it more domain-invariant. A notable benefit here is that the only data we need from the target domain is unlabeled data.
To ensure domain-invariant representation learning, ADA uses adversarial training. Assume that we have a labeled source domain dataset  $D^s$  with examples  $\{(x_1^s,e_1^s),\dots,(x_n^s,e_n^s)\}$ , where  $x_{i}^{s}$  is the token sequence and  $e_i^s$  is the sequence of event tags. We construct auxiliary dataset  $D^a$  with examples
Figure 1: Adversarial Domain Adaptation Framework for Event Trigger Identification
$\{(x_1^a,d_1^a),\ldots ,(x_n^a,d_n^a)\}$  , where  $x_{i}^{a}$  is the token sequence and  $d_i^a$  is the domain label, using token sequences from  $D^s$  and unlabeled target domain sentences. The representation learner  $R$  maps a token sequence  $x_{i} = (x_{i1},\dots,x_{ik})$  into token representations  $h_i = (h_{i1},\dots,h_{ik})$  . The event classifier  $E$  maps representations  $h_i = (h_{i1},\dots,h_{ik})$  to event tags  $e_i = (e_{i1},\dots,e_{ik})$  . The domain predictor  $D$  creates a pooled representation  $p_i =$ $Pool(h_{i1},\dots,h_{ik})$  and maps it to domain label  $d_i^a$  Given this setup, we apply an alternating optimization procedure. In the first step, we train the domain predictor using  $D^{a}$  , to optimize the following loss:
$$
\operatorname * {a r g m i n} _ {D} \mathcal {L} (D (h _ {i} ^ {a}), d _ {i} ^ {a})
$$
In the second step, we train the representation learner and event classifier using  $D^{s}$  to optimize the following loss:
$$
\underset {R, E} {\operatorname {a r g m i n}} \left[ \sum_ {k} \left(\mathcal {L} \left(E \left(h _ {i k} ^ {s}\right), e _ {i k} ^ {s}\right)\right) - \lambda \mathcal {L} \left(D \left(h _ {i} ^ {s}\right), d _ {i} ^ {s}\right) \right]
$$
$\mathcal{L}$  refers to the cross-entropy loss and  $\lambda$  is a hyperparameter. In practice, the optimization in the above equation is performed using a gradient reversal layer (GRL) (Ganin and Lempitsky, 2015). A GRL works as follows. During the forward pass, it acts as the identity, but during the backward pass it scales the gradients flowing through by  $-\lambda$ . We apply a GRL  $g_{\lambda}$  before mapping the pooled representation to a domain label using  $D$ . This changes the optimization to:
$$
\underset {R, E} {\operatorname {a r g m i n}} \left[ \mathcal {L} \left(D \left(g _ {\lambda} \left(p _ {i} ^ {s}\right)\right), d _ {i} ^ {s}\right) + \sum_ {k} \mathcal {L} \left(E \left(h _ {i k} ^ {s}\right), e _ {i k} ^ {s}\right) \right]
$$
In our setup, the event classifier and domain predictors are MLP classifiers. For the representation learner, we experiment with several architectures.
2.2 Representation Learner Models
We experiment with the following models:2
LSTM: A unidirectional LSTM over tokens represented using word embeddings.
2Complete implementation details in the appendix
7619
Statistic	LitBank	TimeBank
#Docs	100	183
#Tokens	210,532	80,281
#Events	7849	8103
Event Density	3.73%	10.10%
Table 1: Dataset Statistics
Model	In-Domain	Out-of-Domain
P	R	F1	P	R	F1
LSTM	61.9	61.5	61.7	86.1	17.1	28.5
LSTM-A	61.1	61.6	61.3	89.0	18.9	31.2
BiLSTM	64.5	61.7	63.1	91.8	14.4	24.9
BiLSTM-A	66.1	62.8	64.4	92.9	18.5	30.9
POS	74.1	51.9	61.1	93.5	9.6	17.4
POS-A	69.6	57.7	63.1	92.5	15.2	26.1
BERT	73.5	72.7	73.1	88.1	28.2	42.7
BERT-A	71.9	71.3	71.6	85.0	35.0	49.6
Table 2: Model performance on domain transfer experiments from LitBank to TimeBank. Presence of the -A suffix indicates that the model uses adversarial training.
BiLSTM: A bidirectional LSTM over word embeddings to incorporate both left and right context.
POS: A BiLSTM over token representations constructed by concatenating word embeddings with embeddings corresponding to part-of-speech tags. This model explicitly introduces syntax.
BERT: A BiLSTM over contextual token representations extracted using BERT (Devlin et al., 2019), similar to the best-performing model on LitBank, reported by Sims et al. (2019).
In our experiments, we use the following datasets:3
- LitBank (Sims et al., 2019): 100 English literary texts with entity and event annotations.
- TimeBank (Pustejovsky et al., 2003): 183 English news articles containing annotations for events and temporal relations between them.
Both datasets follow similar guidelines for event annotation, with an important distinction: LitBank does not annotate events which have not occurred (eg: future, hypothetical or negated events). To overcome this gap, we remove all such events from TimeBank using available metadata about event modality and tense. Table 1 provides a brief overview of statistics for both datasets.
Model	In-Domain	Out-of-Domain
P	R	F1	P	R	F1
LSTM	70.7	78.4	74.4	23.5	75.2	35.8
LSTM-A	69.3	87.5	77.3	25.6	72.9	37.9
BiLSTM	75.4	76.3	75.9	27.6	68.8	39.4
BiLSTM-A	74.2	79.4	76.7	26.3	72.0	38.6
POS	77.4	81.1	79.2	26.4	79.8	39.6
POS-A	76.4	83.0	79.6	27.3	81.9	40.9
BERT	79.6	84.3	81.9	28.1	84.8	42.2
BERT-A	79.8	85.6	82.6	30.3	80.8	44.1
Table 3: Model performance on domain transfer experiments from TimeBank to LitBank. Presence of the -A suffix indicates that the model uses adversarial training.
Tables 2 and 3 present the results of our experiments. Table 2 shows the results when transferring from LitBank to TimeBank while Table 3 presents transfer results in the other direction. From Table 2 (transfer from LitBank to TimeBank), we see that ADA improves out-of-domain performance for all models, by 6.08 F1 on average. BERT-A performs best, reaching an F1 score of 49.6, using no labeled news data. Transfer experiments from TimeBank to LitBank (Table 3) showcase similar trends, with only BiLSTM not showing improvement with ADA. For other models, ADA results in an average out-of-domain F1 score improvement of 1.77. BERT-A performs best, reaching an F1 score of 44.1. We also note that models transferred from LitBank to TimeBank have high precision, while models transferred in the other direction have high recall. We believe this difference stems from the disparity in event density across corpora (Table 1). Since event density in LitBank is much lower, models transferred from LitBank tend to be slightly conservative (high precision), while models transferred from TimeBank are less so (high recall).
When transferring from LitBank to TimeBank, LSTM generalizes better than BiLSTM, which may be because BiLSTM has twice as many parameters making it more prone to overfitting. ADA gives a higher F1 boost with BiLSTM, indicating that it may be acting as a regularizer. Another interesting result is the poor performance of POS when transferring from LitBank to TimeBank. This might stem from the Stanford CoreNLP tagger (trained on news data) producing inaccurate tags for LitBank. Hence using automatically generated POS tags while training on LitBank does not produce
3Unlike prior work, we cannot use the ACE-2005 dataset since it tags specific categories of events, whereas we focus on tagging all possible events.
Category % Example
TimeBank Improvements
Finance	54	the accord was unanimously approved
Political	12	the ukrainian parliament has already ratified it
Reporting	10	from member station kqed , auncil martinez reports
Law	10	mr. antar was charged last month in a civil suit
LitBank Improvements
Archaic	6 his countenance became intolerably fervid
Animal Actions	6 the dogs left off barking, and ran about every way
Human Actions	18 a nod was the answer
Literary	14 there strikes the ebony clock
Table 4: Categorization of TimeBank and LitBank examples on which ADA shows improvement. Words in bold indicate events missed by BERT, but captured by BERT-A.
reliable POS embeddings.
On average, ADA makes supervised models more robust on out-of-domain data, with an average F1 score improvement of 3.9, at no loss of in-domain performance.
What cases does ADA improve on? To gain more insight into the improvements observed on using ADA, we perform a manual analysis of out-of-domain examples that BERT labels incorrectly, but BERT-A gets right. We carry out this analysis on 50 examples from TimeBank and LitBank each. We observe that an overwhelming number of cases from TimeBank use vocabulary in contexts unique to news (43/50 or  $86\%$ ). This includes examples of financial events, political events and reporting events that are rarer in literature, indicating that ADA manages to reduce event extraction models' reliance on lexical features. We make similar observations for LitBank though the proportion of improvement cases with literature-specific vocabulary is more modest (22/50 or  $44\%$ ). These cases include examples with archaic vocabulary, words that have a different meaning in literary contexts and human/ animal actions, which are not common in news. Table 4 presents a detailed breakdown of
Figure 2: Improvement in model performance when finetuning on labeled training data from TimeBank
Figure 3: Improvement in model performance when finetuning on labeled training data from LitBank
these cases, along with examples.4
4 Incorporating Minimal Labeled Data
Finetuning on labeled data: We run finetuning experiments to study improvement in model performance on incorporating small amounts of labeled target domain data. For both domains, we finetune BERT-A, slowly increasing the percentage of labeled data used from  $1\% - 5\%$ .<sup>5</sup> We compare BERT-A with two other models. The first model is naive BERT with no domain adaptation (BERT-NoDA). The second model is a BERT model trained via supervised domain adaptation (BERT-FEDA), which we use as an indicator of ceiling performance. The supervised domain adaptation method we use is the neural modification of frustratingly easy domain adaptation developed in Kim et al. (2016). Frustratingly easy domain adaptation (Daumé III, 2007) uses a feature augmentation strategy to improve performance when annotated data from both source and target domains is available. This algorithm simply duplicates input features 3 times,
4This table does not include generic improvement cases (i.e. no domain-specific vocabulary used), which formed  $14\%$  and  $56\%$  of improvement cases in TimeBank and LitBank.
5We run these experiments 5 times with different random subsets and average performance across all runs.
7621
Dataset	P	R	F1
TimeBank	68.9	65.5	67.2
LitBank	40.3	71.5	51.5
Table 5: Model performance on both domains in the self-training paradigm
creating a source-specific, target-specific and general version of each feature. For source data, only the source-specific and general features are active, while only the target-specific and general features are active for target data. The neural modification works by duplicating the feature extractor module, which is the BiLSTM in our case.
Figures 2 and 3 present the results of these experiments. Performance of all models steadily improves with more data, but BERT-A starts with a much higher F1 score than BERT-NoDA, demonstrating that ADA boosts performance when little annotated training data is available. Performance increase of BERT-NoDA is surprisingly rapid, especially on LitBank. However, it is worth noting that  $5\%$  of the LitBank training set is  $\sim 10,000$  tokens, which is a substantial amount to annotate. Therefore, BERT-A beats BERT-NoDA on sample efficiency. We can also see that BERT-A does not do much worse than BERT-FEDA, which performs supervised adaptation.
Using BERT-A to provide weak supervision: We run further experiments to determine whether finetuned BERT-A can be leveraged for self-training (Yarowsky, 1995; Riloff and Wiebe, 2003). Self-training creates a teacher model from labeled data, which is then used to label a large amount of unlabeled data. Both labeled and unlabeled datasets are jointly used to train a student model. Algorithm 1 gives a quick overview of our self-training procedure. We use  $1\%$  of the training data as  $\mathcal{D}^l$ , with the remaining  $99\%$  used as  $\mathcal{D}^u$ . BERT-A acts as  $\mathcal{T}$ , while  $\mathcal{S}$  is a vanilla BERT model. Table 5 shows the results of self-training on both domains. Self-training improves model performance by nearly 7 F1 points on average. Increase on Time-Bank is much higher which may be due to the high precision-low recall tendency of the teacher model.
In this work, we tackled the task of building generalizable supervised event trigger identification models using adversarial domain adaptation (ADA)
Algorithm 1 SelfTrain  $(\mathcal{D}^l,\mathcal{D}^u,\mathcal{T})$
Input: Teacher Model  $\mathcal{T}$ , Labeled Data  $\mathcal{D}^l = \{(x_1^l, e_1^l), \ldots, (x_m^l, x_m^l)\}$ , Unlabeled Data  $\mathcal{D}^u = \{x_1^u, \ldots, x_n^u\}$ ,
Output: Trained Student Model  $S$
1: Finetune the teacher model  $\mathcal{T}$  by minimizing cross-entropy loss on labeled data
$$
\frac {1}{m} \sum_ {i = 1} ^ {m} \mathcal {L} \left(\mathcal {T} \left(x _ {i} ^ {l}\right), e _ {i} ^ {l}\right)
$$
2: Generate labels  $\{e_1^u,\dots,e_n^u\}$  for unlabeled data  $\mathcal{D}^u$  using  $\mathcal{T}$
3: Train a student model  $\mathcal{S}$  by minimizing cross-entropy loss on both datasets  $\mathcal{D}^l, \mathcal{D}^u$
$$
\frac {1}{m} \sum_ {i = 1} ^ {m} \mathcal {L} (\mathcal {S} (x _ {i} ^ {l}), e _ {i} ^ {l}) + \frac {1}{n} \sum_ {i = 1} ^ {n} \mathcal {L} (\mathcal {S} (x _ {i} ^ {u}), e _ {i} ^ {u})
$$
4: Iterative training: Repeat step 2 using updated student model  $S$
to introduce domain-invariance. Our experiments with two domains (English literature and news) showed that ADA made supervised models more robust on out-of-domain data, with an average F1 score improvement of 3.9. Our best performing model (BERT-A) was able to reach 44-49 F1 across both domains using no labeled target domain data. Preliminary experiments showed that finetuning BERT-A on  $1\%$  labeled data, followed by self-training led to substantial improvement, reaching 51.5 and 67.2 F1 on literature and news respectively. While these results are encouraging, we are yet to match supervised in-domain model performance. Future directions to explore include incorporating noise-robust training procedures (Goldberger and Ben-Reuven, 2017) and example weighting (Dehghani et al., 2018) during self-training, and exploring lexical alignment methods from literature on learning cross-lingual embeddings.
This work was supported by the University of Pittsburgh Medical Center (UPMC) and Abridge AI Inc through the Center for Machine Learning and Health at Carnegie Mellon University. The authors would like to thank the anonymous reviewers for their helpful feedback on this work.
7622
Jun Araki and Teruko Mitamura. 2018. Open-domain event detection using distant supervision. In Proceedings of the 27th International Conference on Computational Linguistics, pages 878-891, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Nathanael Chambers, Taylor Cassidy, Bill McDowell, and Steven Bethard. 2014. Dense event ordering with a multi-pass architecture. Transactions of the Association for Computational Linguistics, 2:273-284.
Xilun Chen, Yu Sun, Ben Athiwaratkun, Claire Cardie, and Kilian Weinberger. 2018. Adversarial deep averaging networks for cross-lingual sentiment classification. Transactions of the Association for Computational Linguistics, 6:557-570.
Yubo Chen, Shulin Liu, Xiang Zhang, Kang Liu, and Jun Zhao. 2017. Automatically labeled data generation for large scale event extraction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 409-419, Vancouver, Canada. Association for Computational Linguistics.
Naomi Daniel, Dragomir Radev, and Timothy Allison. 2003. Sub-event based multi-document summarization. In Proceedings of the HLT-NAACL 03 Text Summarization Workshop, pages 9-16.
Hal Daumé III. 2007. Frustratingly easy domain adaptation. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 256-263, Prague, Czech Republic. Association for Computational Linguistics.
Mostafa Dehghani, Arash Mehrjou, Stephan Gouws, Jaap Kamps, and Bernhard Scholkopf. 2018. Fidelity-weighted learning. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
George Doddington, Alexis Mitchell, Mark Przybocki, Lance Ramshaw, Stephanie Strassel, and Ralph Weischedel. 2004. The automatic content extraction (ACE) program - tasks, data, and evaluation. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC'04), Lisbon, Portugal. European Language Resources Association (ELRA).