SlowGuess/ACL_2020_OCR huggingface dataset:
Bill Yuchen Lin†, Dong-Ho Lee†*, Ming Shen†, Ryan Moreno†, Xiao Huang†, Prashant Shiralkar‡, Xiang Ren†
†University of Southern California
{yuchen.lin, dongho.lee, shenming, morenor}@usc.edu,
{huan183, xiangren}@usc.edu, shiralp@amazon.com
Training neural models for named entity recognition (NER) in a new domain often requires additional human annotations that are usually expensive and time-consuming to collect. Thus, a crucial research question is how to obtain supervision in a cost-effective way. In this paper, we introduce "entity triggers," an effective proxy of human explanations for facilitating label-efficient learning of NER models. An entity trigger is defined as a group of words in a sentence that helps to explain why humans would recognize an entity in the sentence. We crowd-sourced 14k entity triggers for two well-studied NER datasets<sup>1</sup>. Our proposed model, Trigger Matching Network, jointly learns trigger representations and soft matching module with self-attention such that can generalize to unseen sentences easily for tagging. The framework is significantly more cost-effective than the traditional frameworks.
Named entity recognition (NER) is a fundamental information extraction task that focuses on extracting entities from a given text and classifying them using pre-defined categories (e.g., persons, locations, organizations) (Nadeau and Sekine, 2007). Recent advances in NER have primarily focused on training neural network models with an abundance of human annotations, yielding state-of-the-art results (Lample et al., 2016). However, collecting human annotations for NER is expensive and time-consuming, especially in social media messages (Lin et al., 2017a) and technical domains such as biomedical publications, financial documents, legal reports, etc. As we seek to advance NER into more domains with less human effort,
Figure 1: We show two individual entity triggers:  $t_1$  ("had...lunch at") and  $t_2$  ("where the food"). Both are associated to the same entity mention "Rumble Fish" (starting from 7th token) typed as restaurant (RES).
how to learn neural models for NER in a cost-effective way becomes a crucial research problem.
The standard protocol for obtaining an annotated NER dataset involves an annotator selecting token spans in a sentence as mentions of entities, and labeling them with an entity type. However, such annotation process provides limited supervision per example. Consequently, one would need large amount of annotations in order to train high-performing models for a broad range of entity types, which can clearly be cost-prohibitive. The key question is then how can we learn an effective NER model in presence of limited quantities of labeled data?
We, as humans, recognize an entity within a sentence based on certain words or phrases that act as cues. For instance, we could infer that 'Kasdfrcxzv' is likely to be a location entity in the sentence "Tom traveled a lot last year in Kasdfrcxzv." We recognize this entity because of the cue phrase "travel ... in," which suggests there should be a location entity following the word 'in'. We call such phrases "entity triggers." Similar to the way these triggers guide our recognition process, we hypothesize that they can also help the model to learn to generalize efficiently.
Specifically, we define an "entity trigger" (or
<sup>1</sup>The code, data, and a longer version of the paper are at
http://github.com/INK-USC/TriggerNER
8503
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8503-8511
trigger for simplicity) as a group of words that can help explain the recognition process of a particular entity in the same sentence. For example, in Figure 1, "had ... lunch at"² and "where the food" are two distinct triggers associated with the RESTAURANT entity "Rumble Fish." An entity trigger should be a necessary and sufficient cue for humans to recognize its associated entity even if we mask the entity with a random word. Thus, unnecessary words such as "fantastic" should not be considered part of the entity trigger.
In this paper, we argue that a combination of entity triggers and standard entity annotations can enhance the generalization power of NER models. This approach is more powerful because unlabeled sentences, such as "Bill enjoyed a great dinner with Alice at Zcxbz.", can be matched with the existing trigger "had ... lunch at" via their semantic relatedness. This makes it easier for a model to recognize "Zcxbz" as a RESTAURANT entity. In contrast, if we only have the entity annotation itself (i.e., "Rumble Fish") as supervision, the model will require many similar examples in order to learn this simple pattern.
We hypothesize that using triggers as additional supervision is a more cost-effective way to train models. We crowd-sourced annotations of 14,708 triggers on two well-studied NER datasets to study their usefulness for the NER task. Also, we propose a novel framework named Trigger Matching Network that learns trigger representations indicative of entity types during the training phase, and identifies triggers in an unlabeled sentence at inference time to guide a traditional entity tagger for delivering better overall NER performance. Different from conventional training, our learning process has two stages, where the first stage comprises jointly training a trigger classifier and the semantic trigger matcher, followed by a second stage that leverages the trigger representation and the encoding of the given sentence using an attention mechanism to learn a tagger. Experiments show that the proposed model using only  $20\%$  of the trigger-annotated sentences results in a comparable performance as using  $70\%$  of conventional annotated sentences.
We consider the problem of how to cost-effectively learn a model for NER using entity triggers. In this
section, we introduce basic concepts and their notations, present the conventional data annotation process for NER, and provide a formal task definition for learning using entity triggers.
In the conventional setup for supervised learning for NER, we let  $\mathbf{x} = [x^{(1)}, x^{(2)}, \dots, x^{(n)}]$  denote a sentence in the labeled training corpus  $\mathcal{D}_L$ . Each labeled sentence has a NER-tag sequence  $\mathbf{y} = [y^{(1)}, y^{(2)}, \dots, y^{(n)}]$ , where  $y^{(i)} \in \mathcal{V}$  and  $\mathcal{V}$  can be  $\{\mathrm{O}, \mathrm{B - PER}, \dots\}$ . Thus, we have  $\mathcal{D}_L = \{(\mathbf{x_i}, \mathbf{y_i})\}$ , and an unlabeled corpus  $\mathcal{D}_U = \{\mathbf{x_i}\}$ .
We propose to annotate entity triggers in sentences. We use  $T(\mathbf{x},\mathbf{y})$  to represent the set of annotated entity triggers, where each trigger  $t_i \in T(\mathbf{x},\mathbf{y})$  is associated with an entity index  $e$  and a set of word indices  $\{w_i\}$ . Note that we use the index of the first word of an entity as its entity index. That is,  $t = (\{w_1,w_2,\dots \} \to e)$ , where  $e$  and  $w_{i}$  are integers in the range of  $[1,|\mathbf{x}|]$ .
Adding triggers creates a new form of data  $\mathcal{D}_T = \{(\mathbf{x_i},\mathbf{y_i},T(\mathbf{x_i},\mathbf{y_i})\}$ . Our goal is to learn a model for NER from a trigger-labeled dataset  $\mathcal{D}_T$ , such that we can achieve comparable learning performance to a model with a much larger  $\mathcal{D}_L$ .
3 Trigger Matching Networks
We propose a straightforward yet effective framework, named Trigger Matching Networks (TMN), consisting of a trigger encoder (TrigEncoder), a semantic-based trigger matching module (TrigMatcher), and a base sequence tagger (SeqTagger). We have two learning stages for the framework: the first stage (Section 3.1) jointly learns the TrigEncoder and TrigMatcher, and the second stage (Section 3.2) uses the trigger vectors to learn NER tag labels.
3.1 Trigger Encoding & Semantic Matching
Learning trigger representations and semantically matching them with sentences are inseparable tasks. Desired trigger vectors capture the semantics in a shared embedding space with token hidden states, such that sentences and triggers can be semantically matched. Learning an attention-based matching module between entity triggers and sentences is necessary so that triggers and sentences can be semantically matched.
Specifically, for a sentence  $\mathbf{x}$  with multiple entities  $\{e_1,e_2,\dots \}$ , for each entity  $e_i$  we assume that there is a set of triggers  $T_{i} = \{t_{1}^{(i)},t_{2}^{(i)},\dots \}$  without loss of generality. To enable more efficient
Note that a trigger can be a discontinuous phrase.
8504
Figure 2: Two-stage training in Trigger Matching Network (Left). We first jointly train TrigEncoder (via trigger classification) and TrigMatcher (via contrastive loss). Then, we reuse the training data trigger vectors as attention queries in SeqTagger. The inference process (Right). It uses the TrigMatcher to retrieve the  $k$  nearest triggers and average their trigger vectors as the attention query for the trained SeqTagger. Thus, an unseen cue phrase (e.g., "head of... team") can be matched with a seen trigger (e.g., "leader of... group").
batch-based training, we reformat the trigger-based annotated dataset  $\mathcal{D}_T$  such that each new sequence contains only one entity and one trigger. We then create a training instance by pairing each entity with one of its triggers, denoted  $(\mathbf{x},e_i,t_j^{(i)})$
For each reformed training instance  $(\mathbf{x},e,t)$ , we first apply a bidirectional LSTM (BLSTM) on the sequence of word vectors of  $\mathbf{x}$ , obtaining a sequence of hidden states that are the contextualized word representations  $\mathbf{h}_i$  for each token  $x_{i}$  in the sentence. We use  $\mathbf{H}$  to denote the matrix containing the hidden vectors of all of the tokens, and we use  $\mathbf{Z}$  to denote the matrix containing the hidden vectors of all trigger tokens inside the trigger  $t$ .
In order to learn an attention-based representation of both triggers and sentences, we follow the self-attention method introduced by (Lin et al., 2017b) as follows:
$$
\begin{array}{l} \vec {a} _ {s e n t} = \operatorname {S o f t M a x} \left(W _ {2} \tanh \left(W _ {1} \mathbf {H} ^ {T}\right)\right) \\ \mathbf {g} _ {\mathbf {s}} = \vec {\sigma} _ {s e n t} \mathbf {H} \\ \end{array}
$$
$$
\begin{array}{l} \vec {a} _ {t r i g} = \operatorname {S o f t M a x} \left(W _ {2} \tanh  \left(W _ {1} \mathbf {Z} ^ {T}\right)\right) \\ \mathbf {g} _ {\mathbf {t}} = \vec {a} _ {t r i g} \mathbf {Z} \\ \end{array}
$$
$W_{1}$  and  $W_{2}$  are two trainable parameters for computing self-attention score vectors  $\vec{a}_{sent}$  and  $\vec{a}_{trig}$ . We obtain a vector representing the weighted sum of the token vectors in the entire sentence as the final sentence vector  $\mathbf{g_s}$ . Similarly,  $\mathbf{g_t}$  is the final trigger vector, representing the weighted sum of the token vectors in the trigger.
We want to use the type of the associated entity as supervision to guide the trigger representation. Thus, the trigger vector  $\mathbf{g_t}$  is further fed into a multi-class classifier to predict the type of the associated entity  $e$  (such as PER, LOC, etc) which we use  $\mathrm{type}(e)$  to denote. The loss of the trigger
classification is as follows:
$$
L _ {T C} = - \sum \log \mathrm {P} \left(\mathrm {t y p e} (e) \mid \mathbf {g} _ {\mathbf {t}}; \theta_ {T C}\right),
$$
where  $\theta_{TC}$  is a model parameter to learn.
Towards learning to match triggers and sentences based on attention-based representations, we use contrastive loss (Hadsell et al., 2006). The intuition is that similar triggers and sentences should have close representations (i.e., have a small distance between them,  $d$ ). We create negative examples (i.e., mismatches) for training by randomly mixing the triggers and sentences, because TrigMatcher needs to be trained with both positive and negative examples of the form (sentence, trigger, label). For the negative examples, we expect a margin  $m$  between their embeddings. The contrastive loss of soft matching is as follows, where  $\mathbb{1}_{\mathrm{matched}}$  is 1 if the trigger was originally in this sentence and 0 if they are not:
$$
\begin{array}{l} d = \left\| \mathbf {g} _ {\mathbf {s}} - \mathbf {g} _ {\mathbf {t}} \right\| _ {2} \\ L _ {S M} = (1 - \mathbb {1} _ {\mathrm {m a t c h e d}}) \frac {1}{2} (d) ^ {2} \\ + \mathbb {1} _ {\text {m a t c h e d}} \frac {1}{2} \left\{\max  (0, m - d) \right\} ^ {2} \\ \end{array}
$$
The joint loss of the first stage is thus  $L = L_{TC} + \lambda L_{SM}$ , where  $\lambda$  is a hyper-parameter to tune.
3.2 Trigger-Enhanced Sequence Tagging
The learning objective in this stage is to output the tag sequence  $\mathbf{y}$ . Following the most common design of neural NER architecture, BLSTM-CRF (Ma and Hovy, 2016), we incorporate the entity triggers as attention queries to train a trigger-enhanced sequence tagger for NER. Note that the BLSTM used in the TrigEncoder and TrigMatcher modules is the same BLSTM we
8505
use in the SeqTagger to obtain  $\mathbf{H}$ , the matrix containing the hidden vectors of all of the tokens. Given a sentence  $\mathbf{x}$ , we use the previously trained TrigMatcher to compute the mean of all the trigger vectors  $\hat{\mathbf{g}}_{\mathrm{t}}$  associated with this sentence. Following the conventional attention method (Luong et al., 2015), we incorporate the mean trigger vector as the query, creating a sequence of attention-based token representations,  $\mathbf{H}'$ .
$$
\begin{array}{l} \vec {\alpha} = \operatorname {S o f t M a x} \left(\boldsymbol {v} ^ {\top} \tanh  \left(U _ {1} \mathbf {H} ^ {T} + U _ {2} \hat {\mathbf {g}} _ {\mathbf {t}} ^ {T}\right) ^ {\top}\right) \\ \mathbf {H} ^ {\prime} = \vec {\alpha} \mathbf {H} \\ \end{array}
$$
$U_{1}, U_{2}$ , and  $v$  are trainable parameters for computing the trigger-enhanced attention scores for each token. Finally, we concatenate the original token representation  $\mathbf{H}$  with the trigger-enhanced one  $\mathbf{H}'$  as the input  $([\mathbf{H};\mathbf{H}'])$  to the final CRF tagger. Note that in this stage, our learning objective is the same as conventional NER, which is to correctly predict the tag for each token.
3.3 Inference on Unlabeled Sentences
When inferring tags on unlabeled sentences, we do not know the sentence's triggers. Instead, we use the TrigMatcher to compute the similarities between the self-attended sentence representations and the trigger representations, using the most suitable triggers as additional inputs to the SeqTagger. Specifically, we have a trigger dictionary from our training data,  $\mathcal{T} = \{t|\left(\cdot ,\cdot ,t\right)\in$ $\mathcal{D}_T\}$  . Recall that we have learned a trigger vector for each of them, and we can load these trigger vectors as a look-up table in memory. For each unlabeled sentence  $\mathbf{x}$  we first compute its self-attended vector  $\mathbf{g}_{\mathbf{s}}$  as we do when training the TrigMatcher. Using L2-norm distances to compute the contrastive loss, we efficiently retrieve the most similar triggers in the shared embedding space of the sentence and trigger vectors.
Then, we calculate  $\hat{\mathbf{g}}_{\mathbf{t}}$ , the mean of the top  $k$  nearest semantically matched triggers, as this serves a proxy to triggers mentioned for the entity type in the labeled data. We then use it as the attention query for SeqTagger, similarly in Sec. 3.2.
In this section, we first discuss how to collect entity triggers, and empirically study the data-efficiency of our proposed framework.
CONLL 03	PER	ORG	MISC	LOC	Total
# of Entities	1,608	958	787	1,781	5,134
# of Triggers	3,445	1,970	2,057	3,456	10,938
Avg. # of Trig. / Ent.	2.14	2.05	2.61	1.94	2.13
Avg. Trig. Length	1.41	1.46	1.4	1.44	1.43
BC5CDR	DISEASE	CHEMICAL	Total
# of Entities	906	1,085	1,991
# of Triggers	2,130	1,640	3,770
Avg. # of Trig. / Ent.	2.35	1.51	1.89
Avg. Trig. Length	2.00	1.99	2.00
Table 1: Statistics of the crowd-sourced triggers.
4.1 Annotating Entity Triggers
We use a general domain dataset CoNLL2003 (Tjong Kim Sang and De Meulder, 2003) and a bio-medical domain dataset BC5CDR (Li et al., 2016). Both datasets are well-studied and popular in evaluating the performance of neural named entity recognition models such as BLSTM-CRF (Ma and Hovy, 2016).
In order to collect the entity triggers from human annotators, we use Amazon SageMaker Ground Truth<sup>3</sup> to crowd-source entity triggers. More recently, Lee et al. (2020) developed an annotation framework, named LEAN-LIFE, which supports our proposed trigger annotating. Specifically, we sample  $20\%$  of each training set as our inputs, and then reform them (Section 2). Annotators are asked to annotate a group of words that would be helpful in typing and/or detecting the occurrence of a particular entity in the sentence. We masked the entity tokens with their types so that human annotators are more focused on the nonentity words in the sentence when considering the triggers. We consolidate multiple triggers for each entity by taking the intersection of the three annotators' results. Statistics of the final curated triggers are summarized in Table 1.
4.2 Base model
We require a base model to compare with our proposed TMN model in order to validate whether the TMN model effectively uses triggers to improve model performance in a limited label setting. We choose the CNN-BLSTM-CRF (Ma and Hovy, 2016) as our base model for its wide usage in research of neural NER models and applications. Our TMNs are implemented within the same codebase and use the same external word
<sup>3</sup>An advanced version of Amazon Mechanical Turk. https://aws.amazon.com/sagemaker/
8506
CONLL 2003
BLSTM-CRF	TMN	TMN + S.T.
sent.	F1	trig.	F1	F1
5%	69.04	3%	75.33	77.68
10%	76.83	5%	80.2	81.57
20%	81.3	7%	82.02	82.43
30%	83.23	10%	83.53	83.53
40%	84.18	13%	84.22	84.33
50%	84.27	15%	85.03	85.38
60%	85.24	17%	85.36	85.52
70%	86.08	20%	86.01	86.5
Table 2: Labor-efficiency study on BLSTM-CRF and TMN. "sent." means the percentage of the sentences (labeled only with entity tags) we use for BLSTM-CRF, while "trig." denotes the percentage of the sentences (labeled with both entity tags and trigger tags) we use for TMN. 'S.T.' stands for self-training.
vectors from GloVE (Pennington et al., 2014). The hyper-parameters of the CNNs, BLSTMs, and CRFs are also the same. This ensures a fair comparison between a typical non-trigger NER model and our trigger-enhanced framework.
4.3 Results and analysis
Labeled data efficiency. We first seek to study the cost-effectiveness of using triggers as an additional source of supervision. Accordingly, we explore the performance of our model and the baseline for different fractions of the training data. The results on the two datasets are shown in Table 2. The full results are shown in Table 3. We can see that by using only  $20\%$  of the trigger-annotated data, TMN model delivers comparable performance as the baseline model using  $50 - 70\%$  traditional training data. The drastic improvement in the model performance obtained using triggers thus justifies the slightly additional cost incurred in annotating triggers.
Self-training with triggers. We also do a preliminary investigation of adopting self-training (Rosenberg et al., 2005) with triggers. We make inferences on unlabeled data and take the predictions with high confidence as the weak training examples for continually training the model. The confidence is computed following the MNLP metric (Shen et al., 2017), and we take top  $20\%$  every epoch. With the self-training method, we further improve the TMN model's F-1 scores by about  $0.5\sim 1.0\%$ .
Annotation time vs. performance. Although it is hard to accurately study the time cost on the crowd-sourcing platform we use, based on our of
Figure 3: The cost-effectiveness study.
fine simulation we argue that annotating both triggers and entities are about 1.5 times ("BLSTM-CRF (x1.5)" longer than only annotating entities. our offline simulation. In Figure 3, The x-axis for BLSTM-CRF means the number of sentences annotated with only entities, while for TMN means the number of sentences tagged with both entities and triggers. In order to reflect human annotators spending 1.5 to 2 times as long annotating triggers and entities as they spend annotating only entities, we stretch the x-axis for BLSTM-CRF. We can clearly see that the proposed TMN outperforms the BLSTM-CRF model by a large margin. Even if we consider the extreme case that tagging triggers requires twice the human effort ("BLSTM-CRF (x2)"), the TMN is still significantly more labor-efficient in terms of F1 scores.
We introduce "entity trigger" as a complementary annotation. We crowdsourced triggers on two mainstream datasets and will release them to the community, and proposed a novel framework TMN which can generalize to unseen sentences easily for tagging named entities.
This research is based upon work supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via Contract No. 2019-19051600007, NSF SMA 18-29268, and Snap research gift. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government.
8507
Yixin Cao, Zikun Hu, Tat-seng Chua, Zhiyuan Liu, and Heng Ji. 2019. Low-resource name tagging learned with weakly labeled data. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 261-270, Hong Kong, China. Association for Computational Linguistics.
R. Hadsell, S. Chopra, and Y. LeCun. 2006. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06), volume 2, pages 1735-1742.
Braden Hancock, Paroma Varma, Stephanie Wang, Martin Bringmann, Percy Liang, and Christopher Ré. 2018. Training classifiers with natural language explanations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1884-1895, Melbourne, Australia. Association for Computational Linguistics.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 260-270, San Diego, California. Association for Computational Linguistics.
Ouyu Lan, Xiao Huang, Bill Yuchen Lin, He Jiang, Liyuan Liu, and Xiang Ren. 2020. Learning to contextually aggregate multi-source supervision for sequence labeling. In Proceedings of Association for Computational Linguistics. (to appear).
Dong-Ho Lee, Rahul Khanna, Bill Yuchen Lin, Jamin Chen, Seyeon Lee, Qinyuan Ye, Elizabeth Boschee, Leonardo Neves, and Xiang Ren. 2020. Leanlife: A label-efficient annotation framework towards learning from explanation. In Proceedings of Association for Computational Linguistics. (to appear).
Jiao Li, Yueping Sun, Robin J. Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J. Mattingly, Thomas C. Wiegers, and Zhiyong Lu. 2016. Biocreative v cdr task corpus: a resource for chemical disease relation extraction. Database : the journal of biological databases and curation, 2016.
Shen Li, Hengru Xu, and Zhengdong Lu. 2018. Generalize symbolic knowledge with neural rule engine. ArXiv, abs/1808.10326.
Bill Y. Lin, Frank Xu, Zhiyi Luo, and Kenny Zhu. 2017a. Multi-channel BiLSTM-CRF model for emerging named entity recognition in social media. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 160-165, Copenhagen, Denmark. Association for Computational Linguistics.