SlowGuess/ACL_2020_OCR huggingface dataset:
Pierre Lison<sup>1</sup>, Jeremy Barnes<sup>2</sup>, Aliaksandr Hubin<sup>1</sup>, and Samia Touileb<sup>2</sup>
$^{1}$ Norwegian Computing Center, Oslo, Norway
$^{2}$  Language Technology Group, University of Oslo, Norway {plison,ahu}@nr.no, {jeremycb,samiat}@ifi.uio.no
Named Entity Recognition (NER) performance often degrades rapidly when applied to target domains that differ from the texts observed during training. When in-domain labelled data is available, transfer learning techniques can be used to adapt existing NER models to the target domain. But what should one do when there is no hand-labelled data for the target domain? This paper presents a simple but powerful approach to learn NER models in the absence of labelled data through weak supervision. The approach relies on a broad spectrum of labelling functions to automatically annotate texts from the target domain. These annotations are then merged together using a hidden Markov model which captures the varying accuracies and confusions of the labelling functions. A sequence labelling model can finally be trained on the basis of this unified annotation. We evaluate the approach on two English datasets (CoNLL 2003 and news articles from Reuters and Bloomberg) and demonstrate an improvement of about 7 percentage points in entity-level  $F_{1}$  scores compared to an out-of-domain neural NER model.
Named Entity Recognition (NER) constitutes a core component in many NLP pipelines and is employed in a broad range of applications such as information extraction (Raiman and Raiman, 2018), question answering (Mollá et al., 2006), document de-identification (Stubbs et al., 2015), machine translation (Ugawa et al., 2018) and even conversational models (Ghazvininejad et al., 2018). Given a document, the goal of NER is to identify and classify spans referring to an entity belonging to pre-specified categories such as persons, organisations or geographical locations.
NER models often rely on convolutional or recurrent neural architectures, sometimes completed
by a CRF layer (Chiu and Nichols, 2016; Lample et al., 2016; Yadav and Bethard, 2018). More recently, deep contextualised representations relying on bidirectional LSTMS (Peters et al., 2018), transformers (Devlin et al., 2019; Yan et al., 2019) or contextual string embeddings (Akbik et al., 2019) have also been shown to achieve state-of-the-art performance on NER tasks.
These neural architectures require large corpora annotated with named entities, such as Ontonotes (Weischedel et al., 2011) or ConLL 2003 (Tjong Kim Sang and De Meulder, 2003). When only modest amounts of training data are available, transfer learning approaches can transfer the knowledge acquired from related tasks into the target domain, using techniques such as simple transfer (Rodriguez et al., 2018), discriminative fine-tuning (Howard and Ruder, 2018), adversarial transfer (Zhou et al., 2019) or layer-wise domain adaptation approaches (Yang et al., 2017; Lin and Lu, 2018).
However, in many practical settings, we wish to apply NER to domains where we have no labelled data, making such transfer learning methods difficult to apply. This paper presents an alternative approach using weak supervision to bootstrap named entity recognition models without requiring any labelled data from the target domain. The approach relies on labelling functions that automatically annotate documents with named-entity labels. A hidden Markov model (HMM) is then trained to unify the noisy labelling functions into a single (probabilistic) annotation, taking into account the accuracy and confusions of each labelling function. Finally, a sequence labelling model is trained using a cross-entropy loss on this unified annotation.
As in other weak supervision frameworks, the labelling functions allow us to inject expert knowledge into the sequence labelling model, which is often critical when data is scarce or non-existent (Hu et al., 2016; Wang and Poon, 2018). New la
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1518-1533 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
belling functions can be easily inserted to leverage the knowledge sources at our disposal for a given textual domain. Furthermore, labelling functions can often be ported across domains, which is not the case for manual annotations that must be reiterated for every target domain.
1. A broad collection of labelling functions for NER, including neural models trained on various textual domains, gazetteers, heuristic functions, and document-level constraints.
2. A novel weak supervision model suited for sequence labelling tasks and able to include probabilistic labelling predictions.
3. An open-source implementation of these labelling functions and aggregation model that can scale to large datasets<sup>1</sup>.
Unsupervised domain adaptation: Unsupervised domain adaptation attempts to adapt knowledge from a source domain to predict new instances in a target domain which often has substantially different characteristics. Earlier approaches often try to adapt the feature space using pivots (Blitzer et al., 2006, 2007; Ziser and Reichart, 2017) to create domain-invariant representations of predictive features. Others learn low-dimensional transformation features of the data (Guo et al., 2009; Glorot et al., 2011; Chen et al., 2012; Yu and Jiang, 2016; Barnes et al., 2018). Finally, some approaches divide the feature space into general and domain-dependent features (Daumé III, 2007). Multi-task learning can also improve cross-domain performance (Peng and Dredze, 2017).
Recently, Han and Eisenstein (2019) proposed domain-adaptive fine-tuning, where contextualised embeddings are first fine-tuned to both the source and target domains with a language modelling loss and subsequently fine-tuned to source domain labelled data. This approach outperforms several strong baselines trained on the target domain of the WNUT 2016 NER task (Strauss et al., 2016).
Aggregation of annotations: Approaches that aggregate annotations from multiples sources have largely concentrated on noisy data from crowd sourced annotations, with some annotators possibly
being adversarial. The Bayesian Classifier Combination approach of Kim and Ghahramani (2012) combines multiple independent classifiers using a linear combination of predictions. Hovy et al. (2013) learn a generative model able to aggregate crowd-sourced annotations and estimate the trustworthiness of annotators. Rodrigues et al. (2014) present an approach based on Conditional Random Fields (CRFs) whose model parameters are learned jointly using EM. Nguyen et al. (2017b) propose a Hidden Markov Model to aggregate crowd-sourced sequence annotations and find that explicitly modelling the annotator leads to improvements for POSTAGging and NER. Finally, Simpson and Gurevych (2019) proposed a fully Bayesian approach to the problem of aggregating multiple sequential annotations, using variational EM to compute posterior distributions over the model parameters.
Weak supervision: The aim of weakly supervised modelling is to reduce the need for hand-annotated data in supervised training. A particular instance of weak supervision is distant supervision, which relies on external resources such as knowledge bases to automatically label documents with entities that are known to belong to a particular category (Mintz et al., 2009; Ritter et al., 2013; Shang et al., 2018). Ratner et al. (2017, 2019) generalised this approach with the Snorkel framework which combines various supervision sources using a generative model to estimate the accuracy (and possible correlations) of each source. These aggregated supervision sources are then employed to train a discriminative model. Current frameworks are, however, not easily adaptable to sequence labelling tasks, as they typically require data points to be independent. One exception is the work of Wang and Poon (2018), which relies on deep probabilistic logic to perform joint inference on the full dataset. Finally, Fries et al. (2017) presented a weak supervision approach to NER in the biomedical domain. However, unlike the model proposed in this paper, their approach relies on an ad-hoc mechanism for generating candidate spans to classify.
The approach most closely related to this paper is Safranchik et al. (2020), which describe a similar weak supervision framework for sequence labelling based on an extension of HMMs called linked hidden Markov models. The authors introduce a new type of noisy rules, called linking rules, to determine how sequence elements should be grouped into spans of same tag. The main differences be
<https://github.com/NorskRegnesentral/weak-supervision-for-NER.
Figure 1: Illustration of the weak supervision approach.
tween their approach and this paper are the linking rules, which are not employed here, and the choice of labelling functions, in particular the document-level relations detailed in Section 3.1.
Ensemble learning: The proposed approach is also loosely related to ensemble methods such bagging, boosting and random forests (Sagi and Rokach, 2018). These methods rely on multiple classifiers run simultaneously and whose outputs are combined at prediction time. In contrast, our approach (as in other weak supervision frameworks) only requires labelling functions to be aggregated once, as an intermediary step to create training data for the final model. This is a non-trivial difference as running all labelling functions at prediction time is computationally costly due to the need to run multiple neural models along with gazetteers extracted from large knowledge bases.
The proposed model collects weak supervision from multiple labelling functions. Each labelling function takes a text document as input and outputs a series of spans associated with NER labels. These outputs are then aggregated using a hidden Markov model (HMM) with multiple emissions (one per labelling function) whose parameters are estimated in an unsupervised manner. Finally, the aggregated labels are employed to learn a sequence labelling model. Figure 1 illustrates this process. The process is performed on documents from the target domain, e.g. a corpus of financial news.
Labelling functions are typically specialised to detect only a subset of possible labels. For instance, a gazetteer based on Wikipedia will only detect mentions of persons, organisations and geographical locations and ignore entities such as dates or percents. This marks a departure from existing aggregation methods, which are originally designed for crowd-sourced data and where annotators are supposed to make use of the full label set. In addition, unlike previous weak supervision approaches,
we allow labelling functions to produce probabilistic predictions instead of deterministic values. The aggregation model described in Section 3.2 directly captures these properties in the emission model associated with each labelling function.
We first briefly describe the labelling functions integrated into the current system. We review in Section 3.2 the aggregation model employed to combine the labelling predictions. The final labelling model is presented in Section 3.3. The complete list of 52 labelling functions employed in the experiments is available in Appendix A.
3.1 Labelling functions
Out-of-domain NER models The first set of labelling functions are sequence labelling models trained in domains from which labelled data is available. In the experiments detailed in Section 4, we use four such models, respectively trained on Ontonotes (Weischedel et al., 2011), CoNLL 2003 (Tjong Kim Sang and De Meulder, 2003)<sup>2</sup>, the Broad Twitter Corpus (Derczynski et al., 2016) and a NER-annotated corpus of SEC filings (Salinas Alvarado et al., 2015).
For the experiments in this paper, all aforementioned models rely on a transition-based NER model (Lample et al., 2016) which extracts features with a stack of four convolutional layers with filter size of three and residual connections. The model uses attention features and a multi-layer perceptron to select the next transition. It is initialised with GloVe embeddings (Pennington et al., 2014) and implemented in Spacy (Honnibal and Montani, 2017). However, the proposed approach does not impose any constraints on the model architecture and alternative approaches based on e.g. contextualised embeddings can also be employed.
Gazetteers As in distant supervision approaches, we include a number of gazetteers from large knowledge bases to identify named entities. Concretely, we use resources from Wikipedia (Geiß et al., 2018), Geonames (Wick, 2015), the Crunchbase Open Data Map, DBPedia (Lehmann et al., 2015) along with lists of countries, languages, nationalities and religious or political groups.
To efficiently search for occurrences of these entities in large text collections, we first convert each knowledge base into a trie data structure. Prefix search is then applied to extract matches (using
2The ConLL 2003 NER model is of course deactivated for the experimental evaluation on ConLL 2003.
both case-sensitive and case-insensitive mode, as they have distinct precision-recall trade-offs).
Heuristic functions We also include various heuristic functions, each specialised in the recognition of specific types of named entities. Several functions are dedicated to the recognition of proper names based on casing, part-of-speech tags or dependency relations. In addition, we integrate a variety of handcrafted functions relying on regular expressions to detect occurrences of various entities (see Appendix A for details). A probabilistic parser specialised in the recognition of dates, times, money amounts, percents, and cardinal/ordinal values (Braun et al., 2017) is also incorporated.
Document-level relations All labelling functions described above rely on local decisions on tokens or phrases. However, texts are not loose collections of words, but exhibit a high degree of internal coherence (Grosz and Sidner, 1986; Grosz et al., 1995) which can be exploited to further improve the annotations.
We introduce one labelling function to capture label consistency constraints in a document. As noted in (Krishnan and Manning, 2006; Wang et al., 2018), named entities occurring multiple times through a document have a high probability of belonging to the same category. For instance, while Komatsu may both refer to a Japanese town or a multinational corporation, a text including this mention will either be about the town or the company, but rarely both at the same time. To capture these non-local dependencies, we define the following label consistency model: given a text span  $e$  occurring in a given document, we look for all spans  $Z_{e}$  in the document that contain the same string as  $e$ . The (probabilistic) output of the labelling function then corresponds to the relative frequency of each label  $l$  for that string in the document:
$$
P _ {\text {d o c - m a j o r i t y} (e)} (l) = \frac {\sum_ {z \in Z _ {e}} P _ {\text {l a b e l} (z)} (l)}{| Z _ {e} |} \tag {1}
$$
The above formula depends on a distribution  $P_{\mathrm{label}(z)}$ , which can be defined on the basis of other labelling functions. Alternatively, a two-stage model similar to (Krishnan and Manning, 2006) could be employed to first aggregate local labelling functions and subsequently apply document-level functions on aggregated predictions.
Another insight from Grosz and Sidner (1986) is the importance of the attentional structure. When
introduced for the first time, named entities are often referred to in an explicit and univocal manner, while subsequent mentions (once the entity is a part of the focus structure) frequently rely on shorter references. The first mention of a person in a given text is for instance likely to include the person's full name, and is often shortened to the person's last name in subsequent mentions. As in Ratinov and Roth (2009), we determine whether a proper name is a substring of another entity mentioned earlier in the text. If so, the labelling function replicates the label distribution of the first entity.
3.2 Aggregation model
The outputs of these labelling functions are then aggregated into a single layer of annotation through an aggregation model. As we do not have access to labelled data for the target domain, this model is estimated in a fully unsupervised manner.
Model We assume a list of  $J$  labelling functions  $\{\lambda_1,\dots \lambda_J\}$  and a list of  $S$  mutually exclusive NER labels  $\{l_1,\ldots l_S\}$ . The aggregation model is represented as an HMM, in which the states correspond to the true underlying labels. This model has multiple emissions (one per labelling function) assumed to be mutually independent conditional on the latent underlying label.
Formally, for each token  $i\in \{1,\dots,n\}$  and labelling function  $j$  we assume a Dirichlet distribution for the probability labels  $P_{ij}$ . The parameters of this Dirichlet are separate vectors  $\alpha_{j}^{s_{i}}\in \mathcal{R}_{[0,1]}^{S}$ , for each of the latent states  $s_i\in \{1,\ldots ,S\}$ . The latent states are assumed to have a Markovian dependence structure between the tokens  $\{1,\dots,n\}$ . This results in the HMM represented by a dependent mixtures of Dirichlet model:
$$
\boldsymbol {P} _ {i j} \left| \boldsymbol {\alpha} _ {j} ^ {s _ {i}} \quad^ {i n d} \quad \text {D i r i c h l e t} \left(\boldsymbol {\alpha} _ {j} ^ {s _ {i}}\right), \right. \tag {2}
$$
$$
p \left(s _ {i} \mid s _ {i - 1}\right) = \operatorname {l o g i t} ^ {- 1} \left(\omega^ {\left(s _ {i}, s _ {i - 1}\right)}\right), \tag {3}
$$
$$
\mathrm {l o g i t} ^ {- 1} \left(\omega^ {(s _ {i}, s _ {i - 1})}\right) = \frac {e ^ {\omega^ {(s _ {i} , s _ {i - 1})}}}{1 + e ^ {\omega^ {(s _ {i} , s _ {i - 1})}}}. \qquad (4)
$$
Here,  $\omega^{(s_i,s_{i - 1})}\in \mathcal{R}$  are the parameters of the transition probability matrix controlling for a given state  $s_{i - 1}$  the probability of transition to state  $s_i$ . Figure 2 illustrates the model structure.
Parameter estimation The learnable parameters of this HMM are (a) the transition matrix between states and (b) the  $\alpha$  vectors of the Dirichlet distribution associated with each labelling function. The
Figure 2: Aggregation model using a hidden Markov model with multiple probabilistic emissions.
transition matrix is of size  $|S| \times |S|$ , while we have  $|S| \times |J| \alpha$  vectors, each of size  $|S|$ . The parameters are estimated with the Baum-Welch algorithm, which is a variant of EM algorithm that relies on the forward-backward algorithm to compute the statistics for the expectation step.
To ensure faster convergence, we introduce a new constraint to the likelihood function: for each token position  $i$ , the corresponding latent label  $s_i$  must have a non-zero probability in at least one labelling function (the likelihood of this label is otherwise set to zero for that position). In other words, the aggregation model will only predict a particular label if this label is produced by least one labelling function. This simple constraint facilitates EM convergence as it restricts the state space to a few possible labels at every time-step.
Prior distributions The HMM described above can be provided with informative priors. In particular, the initial distribution for the latent states can be defined as a Dirichlet based on counts  $\delta$  for the most reliable labelling function:
$$
p \left(s _ {i}\right) \stackrel {{d}} {{=}} \operatorname {D i r i c h l e t} (\delta). \tag {5}
$$
The prior for each row  $k$  of the transition probabilities matrix is also a Dirichlet based on the frequencies of transitions between the observed classes for the most reliable labelling function  $\kappa_{k}$ :
$$
p \left(s _ {i} \mid s _ {i - 1} = k\right) \stackrel {d} {=} \operatorname {D i r i c h l e t} \left(\kappa_ {k}\right). \tag {6}
$$
Finally, to facilitate convergence of the EM algorithm, informative starting values can be specified for the emission model of each labelling function.
Assuming we can provide rough estimates of the recall  $r_{jk}$  and precision  $\rho_{jk}$  for the labelling function  $j$  on label  $k$ , the initial values for the parameters of the emission model are expressed as:
$$
\alpha_ {j k} ^ {s _ {i}} \propto \left\{ \begin{array}{l} r _ {j k}, \text {i f} s _ {i} = k, \\ (1 - r _ {s _ {i} k}) (1 - \rho_ {j k}) \delta_ {k}, \text {i f} s _ {i} \neq k. \end{array} \right.
$$
The probability of observing a given label  $k$  emitted by the labelling function  $j$  is thus proportional to its recall if the true label is indeed  $k$ . Otherwise (i.e. if the labelling function made an error), the probability of emitting  $k$  is inversely proportional to the precision of the labelling function  $j$ .
Decoding Once the parameters of the HMM model are estimated, the forward-backward algorithm can be employed to associate each token marginally with a posterior probability distribution over possible NER labels (Rabiner, 1990).
3.3 Sequence labelling model
Once the labelling functions are aggregated on documents from the target domain, we can train a sequence labelling model on the unified annotations, without imposing any constraints on the type of model to use. To take advantage of the posterior marginal distribution  $\tilde{p}_s$  over the latent labels, the optimisation should seek to minimise the expected loss with respect to  $\tilde{p}_s$ :
$$
\hat {\theta} = \arg \min  _ {\theta} \sum_ {i} ^ {n} \mathbb {E} _ {y \sim \tilde {p} _ {s}} [ \operatorname {l o s s} \left(h _ {\theta} \left(x _ {i}\right), y\right) ] \tag {7}
$$
where  $h_{\theta}(\cdot)$  is the output of the sequence labelling model. This is equivalent to minimising the cross-entropy error between the outputs of the neural model and the probabilistic labels produced by the aggregation model.
We evaluate the proposed approach on two English-language datasets, namely the CoNLL 2003 dataset and a collection of sentences from Reuters and Bloomberg news articles annotated with named entities by crowd-sourcing. We include a second dataset in order to evaluate the approach with a more fine-grained set of NER labels than the ones in CoNLL 2003. As the objective of this paper is to compare approaches to unsupervised domain adaptation, we do not rely on any labelled data from these two target domains.
3The most reliable labelling function was found in our experiments to be the NER model trained on Ontonotes 5.0.
CoNLL 2003 The CoNLL 2003 dataset (Tjong Kim Sang and De Meulder, 2003) consists of 1163 documents, including a total of 35089 entities spread over 4 labels: ORG, PER, LOC and MISC.
Reuters & Bloomberg We additionally crowd annotate 1054 sentences from Reuters and Bloomberg news articles from Ding et al. (2014). We instructed the annotators to tag sentences with the following 9 Ontonotes-inspired labels: PERSON, NORP, ORG, LOC, PRODUCT, DATETIME, PERCENT, MONEY, QUANTITY. Each sentence was annotated by at least two annotators, and a qualifying test with gold-annotated questions was conducted for quality control. Cohen's  $\kappa$  for sentences with two annotators is 0.39, while Krippendorff's  $\alpha$  for three annotators is 0.44. We had to remove QUANTITY labels from the annotations as the crowd results for this label were highly inconsistent.
Ontonotes-trained NER The first baseline corresponds to a neural sequence labelling model trained on the Ontonotes 5.0 corpus. We use here the same model from Section 3.1, which is the single best-performing labelling function (that is, without aggregating multiple predictions).
We also experimented with other neural architectures but these performed similar or worse than the transition-based model, presumably because they are more prone to overfitting on the source domain.