SlowGuess/ACL_2020_OCR huggingface dataset:
Meihan Tong $^{1}$ , Bin Xu $^{1}$ , Shuai Wang $^{2}$ , Yixin Cao $^{3*}$ , Lei Hou $^{1}$ , Juanzi Li $^{1}$  and Jun Xie $^{4}$
$^{1}$ Knowledge Engineering Laboratory, Tsinghua University, Beijing, China
$^{2}$ SLP Group, AI Technology Department, JOYY Inc, China
$^{3}$ National University of Singapore, Singapore
SPPD Group, Tencent Inc, China
tongmeihan@gmail.com, xubin@tsinghua.edu.cn
wangshuai1@yy.com, caoyixin2011@gmail.com
greener2009@gmail.com, lijuanzi@tsinghua.edu.cn
stiffxie@tencent.com
Event Detection (ED) is a fundamental task in automatically structuring texts. Due to the small scale of training data, previous methods perform poorly on unseen/sparsely labeled trigger words and are prone to overfitting densely labeled trigger words. To address the issue, we propose a novel Enrichment Knowledge Distillation (EKD) model to leverage external open-domain trigger knowledge to reduce the in-built biases to frequent trigger words in annotations. Experiments on benchmark ACE2005 show that our model outperforms nine strong baselines, is especially effective for unseen/sparsely labeled trigger words. The source code is released on https://github.com/shuaiwa16/ekd.git.
Event Detection (ED) aims at detecting trigger words in sentences and classifying them into predefined event types, which shall benefit numerous applications, such as summarization (Li et al., 2019) and reading comprehension (Huang et al., 2019). For instance, in S1 of Figure 1, ED aims to identify the word fire as the event trigger and classify its event type as Attack. Mainstream researches (Chen et al., 2015; Liu et al., 2017, 2018b; Liao and Grishman, 2010b; Zhao et al., 2018; Liu et al., 2018a) focus on the second step event type disambiguation via lexical and contextual features. However, it is also crucial to identify trigger words correctly as the preliminary step.
Trigger word identification is a non-trivial task, which suffers from the long tail issue. Take the benchmark ACE2005 as an example: trigger words with frequency less than 5 account for  $78.2\%$  of the
Figure 1: Examples of ED. fire is the densely labeled trigger for Attack event in ACE2005. Hacked and intifada are the unseen/sparsely labeled triggers in the training corpus. The red ones illustrate the triggers identified by open-domain trigger knowledge.
total. The long tail issue makes supervised methods (Li et al., 2013; Yang et al., 2019) prone to overfitting and perform poorly on unseen/sparsely labeled triggers (Lu et al., 2019). Automatically generating more training instances seems to be a solution: expanding more instances by bootstrapping (Ferguson et al., 2018; Zhang et al., 2019; Cao et al., 2019) and expending more data from distantly supervised methods (Chen et al., 2017; Wang et al., 2019a). However, the performance of these methods on unseen/sparsely labeled trigger words is still unsatisfied, as shown in Table 1. We argue that these methods either lead to the homogeneity of the generated corpus, or subject to the low coverage of knowledge base. More importantly, the expanded data itself is unevenly distributed, and we cannot expect to alleviate the long tail problem with built-in bias data.
In the paper, we empower the model with external knowledge called Open-Domain Trigger Knowledge to provide extra semantic support on unseen/sparsely labeled trigger words and improve trigger identification. Open-Domain Trig
5887
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5887-5897
Table 1: F score on unseen/sparsely and densely labeled triggers. DMBERT (Chen et al., 2015) refers to a supervised-only model with dynamic multi-pooling to capture contextual features; BOOTSTRAP (He and Sun, 2017) expands training data via bootstrapping. DGBERT expands training data with Freebase (Chen et al., 2017).
Method	Unseen	Sparse	Dense
\( DMBERT_{sup-only} \)	54.4	72.5	84.1
\( BOOTSTRAP_{semi-sup} \)	56.6	73.6	86.9
\( DGBERT_{distance-sup} \)	54.7	72.8	84.3
ger Knowledge is defined as a prior that specifies which words can trigger events without subject to pre-defined event types and the domain of texts. As shown in S1 of Figure 1, open-domain trigger knowledge can identify that hearing and fire as event triggers, even if hearing does not fit into any pre-defined event types in ACE2005. With open-domain trigger knowledge, we are able to discover unseen/sparsely triggers from the large-scale unlabeled corpus, which will improve the recall in trigger words identification. However, it is challenging to incorporate open-domain trigger knowledge into ED: Triggers identified by open-domain trigger knowledge do not always fit well with indomain labels, and thus can not be directly adopted as the trigger identification result. For example in S4 of Figure 1, open-domain trigger knowledge argues that exploded is the trigger word, while under the labeling rules of ACE2005, intifada is the trigger word.
Specifically, we propose an Enrichment Knowledge Distillation (EKD) model to efficiently distill open-domain trigger knowledge from both labeled and abundant unlabeled corpora. We first apply a light-weight pipeline to equipment unlabeled sentences with trigger knowledge from WordNet. The method is not limited to specific domains, and thus can guarantee the coverage of trigger words. Then, given the knowledge enhanced data as well as ED annotations, we train a teacher model for better performance; meanwhile, a student model is trained to mimic teacher's outputs using data without knowledge enhancement, which conforms to the distribution during inference. We further promote the generalization of the model by adding noise to the inputs of the student model.
We evaluate our model on the ACE2005 ED benchmark. Our method surpasses nine strong baselines, and is especially effective for unseen/sparsely labeled triggers word. Experiments
also show that the proposed EKD architecture is very flexible, and can be conveniently adapted to distill other knowledge, such as entity, syntactic and argument.
- To the best of our knowledge, we are the first to leverage the wealth of the open-domain trigger knowledge to improve ED.
- We propose a novel teacher-student model (EKD) that can learn from both labeled and unlabeled data, so as to improve ED performance by reducing the in-built biases in annotations.
- Experiments on benchmark ACE2005 show that our method surpasses nine strong baselines which are also enhanced with knowledge. Detailed studies show that our method can be conveniently adapted to distill other knowledge, such as entities.
2.1 Event Detection
Traditional feature-based methods exploit both lexical and global features to detect events (Li et al., 2013). As neural networks become popular in NLP (Cao et al., 2018), data-driven methods use various superior DMCNN, DLRNN and PLMEE model (Duan et al., 2017; Nguyen and Grishman, 2018; Yang et al., 2019) for end-to-end event detection. Recently, weakly-supervised methods (Judea and Strube, 2016; Huang et al., 2017; Zeng et al., 2018; Yang et al., 2018) has been proposed to generate more labeled data. (Gabbard et al., 2018) identifies informative snippets of text as expending annotated data via curated training. (Liao and Grishman, 2010a; Ferguson et al., 2018) rely on sophisticated pre-defined rules to bootstrap from the paralleling news streams. (Wang et al., 2019a) limits the data range of adversarial learning to trigger words appearing in labeled data. Due to the long tail issue of labeled data and the homogeneity of the generated data, previous methods perform badly on unseen/sparsely labeled data and turn to overfitting densely labeled data. With open-domain trigger knowledge, our model is able to perceive the unseen/sparsely labeled trigger words from abundant unlabeled data, and thus successfully improve the recall of the trigger words.
5888
Knowledge Distillation, initially proposed by (Hinton et al., 2015), has been widely adopted in NLP to distill external knowledge into the model (Laine and Aila, 2016; Saito et al., 2017; Ruder and Plank, 2018). The main idea is to adopt a student model to learn from a robust pre-trained teacher model. (Lee et al., 2018; Gong et al., 2018) reinforces the connection between teacher and student model by singular value decomposition and the laplacian regularized least squares. (Tarvainen and Valpola, 2017; Huang et al., 2018) stabilize the teacher model by a lazy-updated mechanism to enable student model not susceptible to external disturbances. (Liu et al., 2019) uses an adversarial imitation approach to enhance the learning procedure. Unlike previous methods that relied on golden annotations, our method is able to learn from pseudo labels and effectively extract knowledge from both labeled and unlabeled corpus.
In the section, we introduce the proposed Enrichment Knowledge Distillation (EKD) model, which leverages open-domain trigger knowledge to improve ED. In general, we have a teacher model and a student model. The teacher is fully aware of open-domain trigger knowledge, while the student is not equipped with open-domain trigger knowledge. We make the student model to imitate the teacher's prediction to distill the open-domain trigger knowledge to our model. Figure 2 illustrates the architecture of the proposed EKD model. During training, we first pre-train the teacher model on labeled data, and then force the student model, under the knowledge-absent situation, to generate pseudo labels as good as the teacher model on both labeled and unlabeled data. By increasing the cognitive gap between teacher and student model, the student model has to learn harder.
We first introduce how to collect the open-domain trigger knowledge in Knowledge Collection. We then illustrate how to exploit the labeled data to pre-train the teacher model in Feature Extraction and Event Prediction. Finally, we elaborate on how to force the student model to learn from the teacher model in Knowledge Distillation.
Given the labeled corpus  $L = \{(S_i,Y_i)\}_{i = 1}^{N_L}$  and abundant unlabeled corpus  $U = \{(S_k)\}_{k = N_L + 1}^{N_T}$ ,
our goal is to jointly optimize two objections: 1) maximize the prediction probability  $P(Y_{i}|S_{i})$  on labeled corpus  $L, 2)$  minimize the prediction probability discrepancy between the teacher  $P(Y_{k}^{\prime}|S_{k}^{+})$  and student model  $P(Y_{k}^{\prime}|S_{k}^{-})$  on both  $L$  and  $U$ , where  $N_{T}$  stand for the total number of sentences in both labeled and unlabeled data.  $S^{+}$  and  $S^{-}$  stand for the enhanced and weakened variant of the raw sentence  $S$ , we will explain them in detail in the Section 3.5.  $Y = \{y_{1}, y_{2}, \ldots, y_{n}\}$  stands for the golden event type label, where each  $y \in Y$  belongs to the 33 event types pre-defined in ACE and a "NEGATIVE" event type (Chen et al., 2015; Nguyen et al., 2016; Feng et al., 2018).  $Y'$  is the pseudo label proposed by pre-trained teacher model.
3.2 Knowledge Collection
Open-domain trigger knowledge elaborates whether a word triggers an event from the perspective of word sense. Whether the trigger is densely labeled or unseen/sparsely labeled, open-domain trigger knowledge will identify them without distinction. For instance in S3 in Figure 1, although hacked is a rare word and has not been labeled, judging from word sense, open-domain trigger knowledge successfully identifies hacked as a trigger word.
We adopt a light-weight pipeline method, called Trigger From WordNet (TFW), to collect open-domain trigger knowledge (Araki and Mitamura, 2018).
$$
S ^ {+} = T F W (S) \tag {1}
$$
TFW uses WordNet as the intermediary. It has two steps, 1) disambiguate word into WordNet sense, 2) determine whether a sense triggers an event. For the first step, we adopt IMS (Zhong and Ng, 2010) to disambiguate word into word sense in WordNet (Miller et al., 1990). We obtain the input features by POS tagger and dependency parser in Stanford CoreNLP (Manning et al., 2014). For the second step, we adopt the simple dictionary-lookup approach proposed in (Araki and Mitamura, 2018) to determine whether a sense triggers an event. TFW is not limited to particular domains, which is able to provide unlimited candidate triggers. With the support of the lexical database, TFW has high efficiency and can be applied to large-scale knowledge collection.
Finally, we obtain a total of 733,848 annotated sentences from New York Times (Sandhaus, 2008)
5889
Figure 2: The architecture of the proposed EKD model. Besides the supervised signals, EKD exploits abundant unlabeled data by ensuring the prediction consistency of raw sentence and knowledge-attending sentence.
corpus in the first half of 2007. The total number of triggers is 2.65 million, with an average of 3.6 triggers per sentence.
3.3 Feature Extraction
We adopt BERT to obtain the hidden representation for both labeled and unlabeled sentences. BERT is a pre-trained language representation model, and BERT has achieved SOTA performance on a wide range of tasks, such as question answering and language inference. The powerful capability of BERT has also been demonstrated in ED scenario (Wang et al., 2019a).
Formally, given the raw sentence  $S$  and knowledge-attending sentence  $S_{+}$ , we feed them into BERT respectively, and adopt the sequence output of the last layer as the hidden representation for each word in  $S$  and  $S^{+}$ .
$$
\begin{array}{l} H = B E R T (S) \\ H _ {+} = B E R T \left(S _ {+}\right) \tag {2} \\ \end{array}
$$
3.4 Event Prediction
After obtaining the hidden representation of sentence  $S$ , we adopt a full-connected layer to determine the event type  $Y$  for each word in sentence  $S$ .
We use  $S_{(i)}$  and  $Y_{(i)}$  to denote the  $i$ -th training sentence and its event type in labeled corpus  $L$ . We first transform the hidden representation  $H$  obtained from Section 3.3 to a result vector  $O$ , where  $O_{ijc}$  represents the probability that the  $j$ -th word in  $S_{i}$  belongs to the  $c$ -th event class. And then we normalize  $O$  by the softmax function to obtain the conditional probability.
$$
p \left(Y _ {(i)} \mid S _ {(i)}, \theta\right) = \sum_ {j = 1} ^ {n} \frac {\exp \left(O _ {i j c}\right)}{\sum_ {c = 1} ^ {C} \exp \left(O _ {i j c}\right)} / n \tag {3}
$$
Given the labeled corpus  $L = \{S_{i},Y_{i}\} |_{i = 1}^{N_{L}}$ , the optimization object is defined as:
$$
J _ {L} (\theta) = - \sum_ {i = 1} ^ {N _ {L}} \log p \left(Y _ {(i)} \mid S _ {(i)}, \theta\right) \tag {4}
$$
3.5 Knowledge Distillation
In this section, we distill open-domain trigger knowledge into our model. The main idea is to force the student model, with only raw texts as the input, to generate as good pseudo labels as the teacher model on both labeled and unlabeled data.
Formally, given golden event type  $Y$ , the objective is:
$$
p (Y | S ^ {+} \theta) = p (Y | S ^ {-}, \theta) \tag {5}
$$
where  $p(Y|S^{+}\theta)$  and  $p(Y|S^{-},\theta)$  are the predictions from the teacher and student model respectively.
We share the parameters of the teacher and student model. The input of teacher model  $S^{+}$  is aware of the open-domain trigger knowledge, and the input of student model  $S^{-}$  does not know. We give the detailed construction process of  $S^{+}$  and  $S^{-}$  below.
Knowledge-attending Sentences  $(S^{+})$  We embed the open-domain trigger knowledge into the sentence by Marking Mechanism. Specifically, we introduce two symbols, named  $B$ -TRI and  $E$ -TRI to mark the beginning and ending boundary of triggers identified by open-domain trigger knowledge. Formally, given the raw sentence  $S = \{w_{1}, w_{2}, \ldots, w_{i}, \ldots, w_{n}\}$  and trigger  $w_{i}$  identified by open-domain trigger knowledge, the knowledge-attending sentence is  $S^{+} =$
$\{w_{1},w_{2},\ldots ,B\text{-TRI},w_{i},E\text{-TRI},\ldots ,w_{n}\}$ . Marking mechanism works well for our feature extractor BERT (Soares et al., 2019), which is very flexible in embedding knowledge, and can be conveniently adapted to other types of knowledge without heavily-engineered work.
Note that the newly added symbols are lack of pre-trained embedding in BERT. Random initialization undermines the semantic meaning of the introduced symbols, where  $B$ -TRI indicates the beginning of a trigger, and  $E$ -TRI means the ending. We address the issue by fine-tuning BERT on the annotation sentences in Section 3.2. Specifically, we adopt Masked LM task (Devlin et al., 2018) to exploit surrounding words to learn the semantic representation of the introduced symbols ( $B$ -TRI and  $E$ -TRI) based on the Harris distributional hypothesis (Harris, 1954). The mask word rate is set to 0.15 and the accuracy of masked words achieves  $92.3\%$  after fine-tune.
Knowledge-absent Sentences  $(S^{-})$  To make the student model learn harder from the teacher model, we further disturb the input of student model by randomly masking out triggers identified by open-domain trigger knowledge. In this way, the student model has to judge the event type of trigger word solely based on the surrounding context. Formally, given the raw sentence  $S = \{w_{1}, w_{2}, \ldots, w_{i}, \ldots, w_{n}\}$  and trigger  $w_{i}$  identified by open-domain trigger knowledge, the knowledge-absent sentence is  $S^{-} = \{w_{1}, w_{2}, \ldots, [MASK], \ldots, w_{n}\}$ . The mask words are not randomly selected, but among triggers determined by open-domain trigger knowledge, avoiding the model is optimized only for the non-trigger negative class.
KL-divergence Loss We move the added symbols to the end of the sentence to ensure strict alignment of words in  $S^{+}$  and  $S^{-}$ , and then we minimize the discrepancy between conditional probability  $p(Y|S^{-},\theta)$  and  $p(Y|S^{+}\theta)$  with KL-divergence loss. Given the collection of labeled and unlabeled corpus  $T = \{(S_k)\}_{k=1}^{N_L + N_U}$ , the KL-divergence loss is:
$$
\begin{array}{l} J _ {T} (\theta) = \mathbf {K L} (p (Y | S ^ {+}, \theta) | | p (Y | S ^ {-}, \theta)) \\ = \sum_ {k = 1} ^ {N _ {L} + N _ {U}} p \left(Y _ {(k)} \mid S _ {(k)} ^ {+}, \theta\right) \frac {p \left(Y _ {(k)} \mid S _ {(k)} ^ {+} , \theta\right)}{p \left(Y _ {(k)} \mid S _ {(k)} ^ {-} , \theta\right)} \tag {6} \\ \end{array}
$$
KL divergence is asymmetric in the two distributions. We treat predictions from knowledge-absent
inputs as approximate distributions and predictions from knowledge-attending inputs as approximated distributions. If we reverse the direction of approximation, the experimental results decline significantly. The reason may be that we should ensure the low-confidence predictions approximate the high-confidence predictions.
3.6 Joint Training
The final optimization objection is the integration of the supervised loss from labeled dataset and KL-divergence loss from unlabeled dataset defined in Equation 4 and 6.
$$
J (\theta) = J _ {L} (\theta) + \lambda * J _ {T} (\theta) \tag {7}
$$
We stop the gradient descent of teacher model when calculating  $J_{T}$  to ensure that the learning is from teacher to student.
Since unlabeled data is much larger than the labeled data, joint training leads the model quickly overfitting the limited labeled data while still underfitting the unlabeled data. To handle the issue, we adopt the Training Signal Annealing (TSA) technique proposed in (Xie et al., 2019) to linearly release the 'training signals' of the labeled examples as training progresses.
Datasets For the labeled corpus, we adopt dataset ACE2005 to evaluate the overall performance. ACE2005 contains 13,672 labeled sentences distributed in 599 articles. Besides the pre-defined 33 event types, we incorporate an extra "Negative" event type for non-trigger words. Following (Chen et al., 2015), we split ACE2005 into 529/30/40 for train/dev/test respectively.
Evaluation We report the Precision, Recall and micro-averaged F1 scores in the form of percentage over all 33 events. A trigger is considered correct if both its type and offsets match the annotation.
Hyperparameters For feature extraction, we adopt BERT as our backbone, which has 24 16-head attention layers and 1024 hidden embedding dimension. For the batch size, The batch size of labeled data is 32, and we set the proportion of labeled and unlabeled data to 1:6. For most of our experiments, we set the learning rate 3e-5, the maximum sequence length 128 and the  $\lambda$  in joint training 1. Our model trains on one V100 for a
5891
half day. The best result appears around 12,500 epochs. Balancing the performance and training efficiency, we actually use 40,236 unlabeled data for knowledge distillation unless otherwise stated. All reported results are the average results of ten runs. We use Adam as the gradient descent optimizer.
Baselines As our methods incorporate open-domain trigger knowledge, for fair competition, we compare our methods with two data-driven methods and five state-of-the-art knowledge-enhanced methods, including: DMCNN proposes a dynamic multi-pooling layer above CNN model to improve event detection (Chen et al., 2015). DLRNN exploits document information via recurrent neural networks (Duan et al., 2017). ANN-S2 exploits argument information to improve ED via supervised attention mechanisms (Liu et al., 2017). GMLATT adopts a gated cross-lingual attention to exploit the complement information conveyed by multilingual data (Liu et al., 2018a). GCN-ED exploits structure dependency tree information via graph convolutions networks and entity mention-guided pooling (Nguyen and Grishman, 2018). Lu's DISTILL proposes a -learning approach to distill generalization knowledge to handle overfitting (Lu et al., 2019). TS-DISTILL exploits the entity ground-truth and uses an adversarial imitation based knowledge distillation approach for ED (Liu et al., 2019). AD-DMBERT adopts an adversarial imitation model to expend more training data (Wang et al., 2019b). DRMM employs an alternative dual attention mechanism to effectively integrate image information into ED (Tong et al., 2020). The last two baselines both use BERT as feature extractor.
Table 2: Overall Performance on ACE2005 dataset (\%). The results of baselines are adapted from their original papers.
Method	Precision	Recall	F1
DMCNN	75.6	63.6	69.1
DLRNN	77.2	64.9	70.5
ANN-S2	78.0	66.3	71.7
GMLATT	78.9	66.9	72.4
GCN-ED	77.9	68.8	73.1
Lu's DISTILL	76.3	71.9	74.0
TS-DISTILL	76.8	72.9	74.8
AD-DMBERT	77.9	72.5	75.1
DRMM	77.9	74.8	76.3
EKD (Ours)	79.1	78.0	78.6
Table 2 presents the overall performance of the
proposed approach on ACE2005. As shown in Table 2, EKD (our) outperforms various state-of-the-art models, showing the superiority of open-domain trigger knowledge and the effectiveness of the proposed teacher-student model. BERT-based models AD-DMBERT, DRMM and EKD (ours) significantly outperform the CNN-based or LSTM-based models, which is due to the ability to capture contextual information as well as large scale pretraining of BERT. Compared to these BERT-based models, our methods consistently improves the F score by  $3.5\%$  and  $2.3\%$ , which shows the superiority of our method even if the encoder is powerful enough.
Compared to data-driven methods DMCNN and DLRNN, knowledge enhanced methods Lu's DISTILL, TS-DISTILL and EKD (ours) improve the recall by a large margin. Due to the small scale of ACE2005, it is quite tricky to disambiguate triggers solely based on the surrounding context. Enhanced by external knowledge, these methods have a stand-by commonsense to depend on, which prevents from overfitting densely labeled trigger words and thus can discover more trigger words. Among them, our model achieves the best performance, which may be caused by two reasons: 1) The superiority of open-domain trigger knowledge. Compared to general linguistic knowledge used in Lu's DISTILL and entity type knowledge used in TS-DISTILL, open-domain trigger knowledge is more task-related, which directly provides trigger candidates for trigger identification, and thus is more informative. 2) The superiority of the proposed teacher-student model. Our method is able to learn open-domain trigger knowledge from unlimited unlabeled data, while Lu's DISTILL and TS-DISTILL can only learn from labeled data.