Venue: NAACL 2016

# Name Tagging for Low-resource Incident Languages based on

Expectation-driven Learning

Boliang Zhang\({}^{1}\), Xiaoman Pan\({}^{1}\), Tianlu Wang\({}^{2}\), Ashish Vaswani\({}^{3}\),

**Heng Ji\({}^{1}\), Kevin Knight\({}^{3}\), Daniel Marcu\({}^{3}\) \({}^{1}\) Computer Science Department, Rensselaer Polytechnic Institute {zhangb8,panx2,jih}@rpi.edu \({}^{2}\) Computer Science Department, Zhejiang University \({}^{3}\) Information Sciences Institute, University of Southern California {vaswani,knight,marcu}@isi.edu**

###### Abstract

In this paper we tackle a challenging name tagging problem in an emergent setting - the tagger needs to be complete within a few hours for a new incident language (IL) using very few resources. Inspired by observing how human annotators attack this challenge, we propose a new expectation-driven learning framework. In this framework we rapidly acquire, categorize, structure and zoom in on IL-specific expectations (rules, features, patterns, gazetteers, etc.) from various non-traditional sources: consulting and encoding linguistic knowledge from native speakers, mining and projecting patterns from both mono-lingual and cross-lingual corpora, and typing based on cross-lingual entity linking. We also propose a cost-aware combination approach to compose expectations. Experiments on seven low-resource languages demonstrate the effectiveness and generality of this framework: we are able to setup a name tagger for a new IL within two hours, and achieve 33.8%-65.1% F-score 1.

Footnote 1: The resources developed in this paper, including the survey, patterns and gazetteers, are available at [http://nlp.cs.rpi.edu/data/elisaienaacl16.zip](http://nlp.cs.rpi.edu/data/elisaienaacl16.zip)

_Proceedings of NAACL-HLT 2016_, pages 249–259, San Diego, California, June 12-17, 2016. ©2016 Association for Computational Linguistics

## 1 Introduction: "Tibetan Room"

In many emergent situations such as disease outbreaks and natural disasters, there is great demand to rapidly develop a Natural Language Processing (NLP) system, such as name tagger, for a "surprise" Incident Language (IL) with very few resources. Traditional supervised learning methods that rely on large-scale manual annotations would be too costly.

Let's start by investigating how a human would discover information in a foreign IL environment. When we are in a foreign country, even if we don't know the language, we would still be able to guess the word "_gate_" from the airport broadcast based on its frequency and position in a sentence; guess the word "_station_" by pattern mining of many subway station labels; and guess the word "_left_" or "_right_" from a taxi driver's GPS speaker by matching movement actions. We designed a "_Tibetan Room_" game, similar to "_Chinese Room_" Searle (1980), by asking a human user who doesn't know Tibetan to find persons, locations and organizations from some Tibetan documents. We designed an interface where test sentences are presented to the player one by one. When the player clicks token, the interface will display up to 100 manually labeled Tibetan sentences that include this token. The player can also see translations of some common words and a small gazetteer of common names (800 entries) in the interface.

players who don't know Tibetan joined the game. Their name tagging F-scores ranged from 0% to 94%. We found that good players usually bring in some kind of "_expectations_" derived from their own native languages, or general linguistic knowledge, or background knowledge about the scenario. Then they actively search, confirm, adjust and update these expectations during tagging. For example, they know from English that location names are often ended with suffix words such as "_city_" and "_country_", so they search for phrases starting or ending with the translations of these suffix words. After they successfully tag some seeds, they will continue to discover more names based on more expectations.



For example, if they already tagged an organization name \(A\), and now observe a sequence matching a common English pattern "_[A (Organization)]' s [Title] [B (Person)]"_, they will tag \(B\) as a person name. And if they know the scenario is about Ebola, they will be looking for a phrase with translation similar to "_West Africa_" and tag it as a location. Similarly, based on the knowledge that names appear in a conjunction structure often have the same type, they propagate high-confidence types across multiple names. They also keep gathering and synthesizing common contextual patterns and rules (such as position, frequency and length information) about names and non-names to expand their expectations. For example, after observing a token frequently appearing between a subsidiary and a parent organization, they will predict it as a preposition similar to "_of_" in English, and tag the entire string as a nested organization.

Based on these lessons learned from this game, we propose to automatically acquire and encode expectations about what will appear in IL data (names, patterns, rules), and encode those expectations to drive IL name tagging. We explored various ways of systematically discovering and unifying latent and expressed expectations from nontraditional resources:

* **Language Universals**: Language-independent rules and patterns;
* **Native Speaker**: Interaction with native speakers through a machine-readable survey and supervised active learning;
* **Prior Mining**: IL entity prior knowledge mining from both mono-lingual and cross-lingual corpora and knowledge bases;

Furthermore, in emergent situations these expectations might not be available at once, and they may have different costs, so we need to organize and prioritize them to yield optimal performance within given time bounds. Therefore we also experimented with various **cost-aware** composition methods with the input of acquired expectations, plus a time bound for development (1 hour, 2 hours), and the output as a wall-time schedule that determines the best sequence of applying modules and maximizes the use of all available resources. Experiments on seven low-resource languages demonstrate that our framework can create an effective name tagger for an IL within a couple of hours using very few resources.

## 2 Starting Time: Language Universals

First we use some language universal rules, gazetteers and patterns to generate a binary feature vector \(F=\{f_{1},f_{2},...\}\) for each token. Table 1 shows these features along with examples. An identification rule is \(r_{I}=\langle T_{I},f=\{f_{a},f_{b},...\}\rangle\) where \(T_{I}\) is a "B/I/O" tag to indicate the beginning, inside or outside of a name, and \(\{f_{a},f_{b},...\}\) is a set of selected features. If the features are all matched, the token will be tagged as \(T_{I}\). Similarly, a classification rule is \(r_{C}=\langle T_{C},f=\{f_{a},f_{b},...\}\rangle,\) where \(T_{C}\) is "Person/Organization/Location". These rules are triggered in order, and some examples are as follows: \(\langle\text{B},\)\(\{\)AllUppercased\(\}\rangle\), \(\langle\text{PER},\)\(\{\)PersonGaz\(\}\rangle\), \(\langle\text{ORG},\)\(\{\)Capitalized, LongLength\(\}\rangle\), etc.

## 3 Expectation Learning

### Approach Overview

Figure 1 illustrates our overall approach of acquiring various expectations, by simulating the strategies human players adopted during the Tibetan Room game. Next we will present details about discovering expectations from each source.

### Survey with Native Speaker

The best way to understand a language is to consult people who speak it. We introduce a human-in

Figure 1: Expectation Driven Name Tagger Overview



the-loop process to acquire knowledge from native speakers. To meet the needs in the emergent setting, we design a comprehensive survey that aims to acquire a wide-range of IL-specific knowledge from native speakers in an efficient way. The survey categorizes questions and organizes them into a tree structure, so that the order of questions is chosen based on the answers of previous questions. The survey answers are then automatically translated into rules, patterns or gazetteers in the tagger. Some example questions are shown in Table 2.

### Mono-lingual Expectation Mining

We use a bootstrapping method to acquire IL patterns from unlabeled mono-lingual IL documents. Following the same idea in Agichtein and Gravano (2000); Collins and Singer (1999), we first use names identified by high-confident rules as seeds, and generalize patterns from the contexts of these seeds. Then we evaluate the patterns and apply high-quality ones to find more names as new seeds. This process is repeated iteratively 2.

Footnote 2: We empirically set the number of iterations as 2 in this paper.

We define a pattern as a triple \(\langle left,name,right\rangle\), where \(name\) is a name, left and right3 are context vectors with weighted terms (the weight is computed based on each token's tf-idf score). For example, from a Hausa sentence "_gwannatin kasar Sin ta samar wa kaasshen yammacin Afirka ... (the Government of China has given ... products to the West African countries)_", we can discover a pattern:

Footnote 3: \(left\) and \(right\) are the context three tokens before and after the name

* \(left\): \(\langle\textbf{gwannatin}~{}\text{(governmentment), 0.5}\rangle\), \(\langle\textbf{kasar}~{}\text{(country), 0.6}\rangle\)
* \(name\): \(\langle\textbf{Sin}~{}\text{(China), 0.5}\rangle\)
* \(right\): \(\langle\textbf{ta}~{}\text{(by), 0.2}\rangle\)

This pattern matches strings like "_gwammatin kasar Fiji ta (by the government of Fiji)_".

For any two triples \(t_{i}=\langle l_{i},name_{i},r_{i}\rangle\) and \(t_{j}=\langle l_{j},name_{j},r_{j}\rangle\), we comput e their similarity by:

\[Sim(t_{i},t_{j})=l_{i}\cdot l_{j}+r_{i}\cdot r_{j}\]

We use this similarity measurement to cluster all triples and select the centroid triples in each cluster as candidate patterns.

Similar to Agichtein and Gravano (2000), we evaluate the quality of a candidate pattern P by:

\[Conf(P)=\frac{P_{positive}}{(P_{positive}+P_{negative})}\]

,where \(P_{positive}\) is the number of positive matches for \(P\) and \(P_{negative}\) is the number of negative matches. Due to the lack of syntactic and semantic resources to refine these lexical patterns, we set a conservative confidence threshold 0.9.

### Cross-lingual Expectation Projection

Name tagging research has been done for high-resource languages such as English for over twenty years, so we have learned a lot about them. We collected 1,362 patterns from English name tagging literature. Some examples are listed below:

* \(\langle\{\},\{PER\},\{<say>,<.>\}\rangle\)
* \(\langle\{<headquarter>,<in>\},\{LOC\},\{\}\rangle\)
* \(\langle\{<secretary>,<of>\},\{ORG\},\{\}\rangle\)
* \(\langle\{<in>,<the>\},\{LOC\},\{<area>\}\rangle\)

\begin{table}
\begin{tabular}{p{48.4pt} p{270.0pt}} \hline Features & Examples (Feature name is underlined) \\ \hline in English Gazetter & **-PerGaze**: person (\(472,765\)); **LocGazz**: location (\(211,872\)); **OrggGaz**: organization (\(124,403\)); **Title** (\(889\)); **NoneName** (\(2,380\)). \\ Case & **Capitalized**; **- AllUppercased**; **- MixedCase** \\ \multicolumn{2}{p{48.4pt}}{} \\  & **InternalPeriod**: includes an internal period \\ Digit & **-Digits**: consisted of digits \\ Length & **-LongLength**: a name including more than 4 tokens is likely to be an ORG \\ TF-IDF & **-TF-IDF**: if a capitalized word appears at the beginning of a sentence, and has a low TF-IDF, then it’s unlikely to be a name tree structure, so that the order of questions is chosen based on the answers of previous questions. The survey answers are then automatically translated into rules, patterns or gazetteers in the tagger. Some example questions are shown in Table 2.

 

Besides the static knowledge like patterns, we can also dynamically acquire expected names from topically-related English documents for a given IL document. We apply the Stanford name tagger [14] to the English documents to obtain a list of expected names. Then we translate the English patterns and expected names to IL. When there is no human constructed English-to-IL lexicon available, we derive a word-for-word translation table from a small parallel data set using the GIZA\(++\) word alignment tool [30]. We also convert IL text to Latin characters based on Unicode mapping4, and then apply Soundex code [25, 6] to find the IL name equivalent that shares the most similar pronunciation as each English name. For example, the Bengali name "[\(\overleftarrow{\text{e}}\)i] **Q@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@*l@* l@*l 

\(\phi({\textbf{x}})\), which is a function used to evaluate each instance **x** in the unlabeled pool \(U\). Algorithm 1 illustrates the procedure.

```
1:\(L\leftarrow\text{labeled set}\), \(U\leftarrow\text{unlabeled pool}\)
2:\(\phi(\cdot)\leftarrow\text{query strategy}\), \(B\gets\text{query batch size}\)
3:\(M\leftarrow\text{maximum number of tokens}\)
4:while Length(\(L\))\(<M\)do
5:\(\theta=\text{train}(L)\);
6:for\(b\in\{1,2,...,B\}\)do
7:\({\textbf{x}_{b}^{*}}=\text{arg max}_{x\in U}\phi({\textbf{x}})\)
8:\(L=L\cup\{{\textbf{x}_{b}^{*}},\text{label}({\textbf{x}_{b}^{*}})\}\)
9:\(U=U-{\textbf{x}_{b}^{*}}\)
10:endfor
11:endwhile
```

**Algorithm 1** Pool-based Active Learning

Jing et al. (2004) proposed an entropy measure for active learning for image retrieval task. We compared it with other measures proposed by Settles and Craven (2008) and found that **sequence entropy (SE)** is most effective for our name tagging task. We use \(\phi^{SE}\) to represent how informative a sentence is:

\[\phi^{SE}({\textbf{x}})=-\sum_{t=1}^{T}\sum_{m=1}^{M}P_{\theta}(y_{t}=m)logP_{ b}(y_{t}=m)\]

, where T is the length of **x**, \(m\) ranges over all possible token labels and \(P_{\theta}(y_{t}=m)\) is the probability when \(y_{t}\) is tagged as \(m\).

## 5 Cost-aware Combination

A new requirement for IL name tagging is a **Linguistic Workflow Generator**, which can generate an activity schedule to organize and maximize the use of acquired expectations to yield optimal F-scores within given time bounds. Therefore, the input to the IL name tagger is not only the test data, but also a time bound for development (1 hour, 2 hours, 24 hours, 1 week, 1 month, etc.).

Figure 2 illustrates our cost-aware expectation composition approach. Given some IL documents as input, as the clock ticks, the system delivers name tagging results at time 0 (immediately), time 1 (e.g., in one hour) and time 2 (e.g., in two hours). At time 0, name tagging results are provided by the universal tagger described in Section 2. During the first hour, we can either ask the native speaker to annotate a small amount of data for supervised active learning of a CRFs model, or fill in the survey to build a rule-based tagger. We estimate the confidence value of each expectation-driven rule based on its precision score on a small development set of ten documents. Then we apply these rules in the priority order of their confidence values. When the results of two taggers are conflicting on either mention boundary or type, if the applied rule has high confidence we will trust its output, otherwise adopt the CRFs model's output.

## 6 Experiments

In this section we will present our experimental details, results and observations.

### Data

We evaluate our framework on seven low-resource incident languages: Bengali, Hausa, Tagalog, Tamil, Thai, Turkish and Yoruba, using the ground-truth name tagging annotations from the DARPA LORELEI program 6. Table 3 shows data statistics.

Footnote 6: [http://www.darpa.mit/program/low-resource-languages-for-emergent-incidents](http://www.darpa.mit/program/low-resource-languages-for-emergent-incidents)

### Cost-aware Overall Performance

We test with three checking points: starting time, within one hour, and within two hours. Based on the combination approach described in Section 5, we can have three possible combinations of the expectation-driven learning and supervised active learning methods during two hours: (1) expectation-driven learning + supervised active learning; (2) supervised active learning + expectation-driven learning; and (3) supervised active learning for two hours. Figure 3 compares the overall performance of these combinations for each language.

We can see that our approach is able to rapidly set up a name tagger for an IL and achieves promising performance. During the first hour, there is no clear winner between expectation-driven learning or

\begin{table}
\begin{tabular}{p{42.7pt} p{28.5pt} p{28.5pt} p{28.5pt} p{28.5pt}} \hline \hline
**Language II. Test** & **Name** & **Unique Name** & **IL Dev.** & **IL-English** \\  & **Docs** & **Name Name** & **Does Does** & **Does** \\ \hline Bengali & 100 & 4,713 & 2,820 & 12,495 & 169 \\ Hausa & 100 & 1,619 & 950 & 13,652 & 645 \\ Tagalog & 100 & 6,119 & 3,375 & 1,616 & 145 \\ Tamil & 100 & 4120 & 2,871 & 4,597 & 166 \\ Thai & 100 & 4,954 & 3,314 & 10,000 & 191 \\ Turkish & 100 & 2,694 & 1,323 & 10,000 & 484 \\ Yoruba & 100 & 3,745 & 2,337 & 427 & 252 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Data Statistics 

supervised active learning. But it's clear that supervised active learning for two hours is generally not the optimal solution. Using Hausa as a case study, we take a closer look at the supervised active learning curve as shown in Figure 4. We can see that supervised active learning based on simple lexical features tends to converge quickly. As time goes by it will reach its own upper-bound of learning and generalizing linguistic features. In these cases our proposed expectation-driven learning method can compensate by providing more explicit and deeper IL-specific linguistic knowledge.

### Comparison of Expectation Discovery Methods

Table 4 shows the performance gain of each type of expectation acquisition method. IL gazetteers covered some common names, especially when the universal case-based rules failed at identifying names from non-Latin languages. IL name patterns were mainly effective for classification. For example, the Tamil name "_Gebg@n



Figure 3: Comparison of methods combining expectation-driven learning and supervised active learning given various time bounds



Table 5 presents the detailed break-down scores for all languages. We can see that name identification, especially organization identification is the main bottleneck for all languages. For example, many organization names in Hausa are often very long, nested or all low-cased, such as "_makarantar horas da Malaman makaranta ta Bawa Jan Gwazzo_ (Bawa Jan Gwarzo Memorial Teachers College)" and "_kungiyar masana'antu da tattalin arziki ta kasar Sin (China's Association of Business and Industry)_". Our name tagger will further benefit from more robust universal word segmentation, rich morphology analysis and IL-specific knowledge. For example, in Tamil "\({}_{\text{a}}\)" is a visarga used as a diacritic to write foreign sounds, so we can infer a phrase including it (e.g., "@qualitechio" (Haifa)") is likely to be a foreign name. Therefore our survey should be enriched by exercising with many languages to capture more categories of linguistic phenomena.

## 7 Related Work

Name Tagging is a well-studied problem. Many types of frameworks have been used, including rules [13, 26], supervised models using monolingual labeled data [40, 8, 9, 14, 15], bilingual labeled data [16, 17, 18, 19] or naturally partially annotated data such as Wikipedia [15, 14, 16, 17, 18, 19, 20, 21], and unsupervised learning [13, 14, 22, 23] or naturally partially annotated data such as Wikipedia [15, 16, 17, 18, 19, 20, 24, 25, 26, 27, 28, 29, 30], and unsupervised learning [13, 14, 22, 21, 23].

Name tagging has been explored for many non-English languages such as in Chinese [14, 15, 16, 17], Japanese [15, 16, 18, 19], Arabic [19], Catalan [17, 18, 19], Bulgarian [20], Dutch [2], French [2], Spanish language [21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 188, 189, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 222, 231, 232, 240, 209, 224, 208, 209, 235, 201, 236, 237, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 311, 320, 321, 322, 323, 324, 325, 326, 327, 328, 333, 340, 341, 342, 343, 344, 345, 346, 347, 348, 349, 350, 351, 352, 353, 354, 355, 356, 357, 358, 359, 360, 361, 362, 363, 364, 365, 366, 366, 367, 368, 369, 370, 371, 372, 373, 374, 375, 376, 377, 378, 379, 380, 381, 382, 383, 384, 385, 386, 387, 388, 388, 389, 390, 391, 392, 393, 394, 395, 396, 397, 398, 399, 400, 41, 42, 435, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 66, 67, 68, 69, 70, 72, 74, 75, 76, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 109, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 123, 124, 125, 126, 127, 128, 129, 131, 140, 141, 143, 144, 145, 146, 147, 148, 149, 150, 161, 172, 173, 174, 175, 176, 177, 178, 179, 180, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 203, 204, 205, 206, 207, 208, 209, 210, 209, 211, 223, 209, 224, 209, 235, 207, 209, 211, 236, 209, 237, 238, 209, 240, 250, 209, 261, 209, 270, 281, 293, 294, 209, 211, 238, 209, 239, 241, 209, 251, 263, 264, 265, 266, 272, 281, 293, 295, 296, 297, 298, 299, 300, 310, 321, 323, 324, 325, 326, 327, 328, 329, 330, 320, 321, 328, 329, 333, 329, 340, 341, 343, 345, 346, 347, 348, 349, 350, 351, 353, 356, 357, 359, 360, 371, 378, 379, 380, 392, 393, 395, 396, 397, 398, 399, 400, 41, 42, 435, 446, 447, 448, 453, 461, 470, 481, 490, 400, 41, 42, 435, 447, 449, 454, 462, 471, 481, 490, 400, 41, 449, 401, 449, 402, 449, 403, 404, 405, 406, 407, 408, 409, 410, 42, 449, 404, 409, 411, 449, 412, 449, 425, 449, 436, 447, 449, 455, 462, 471, 481, 490, 403, 40 

et al., 2000), German (Thielen, 1995), Italian (Cucchiarelli et al., 1998), Greek (Karkaletsis et al., 1999), Spanish (Arevalo et al., 2002), Portuguese (Hana et al., 2006), Serbo-croatian (Nenadic and Spasic, 2000), Swedish (Dalianis and Astrom, 2001) and Turkish (Tur et al., 2003). However, most of previous work relied on substantial amount of resources such as language-specific rules, basic tools such as part-of-speech taggers, a large amount of labeled data, or a huge amount of Web ngram data, which are usually unavailable for low-resource ILs. In contrast, in this paper we put the name tagging task in a new emergent setting where we need to process a surprise IL within very short time using very few resources.

The TIDES 2003 Surprise Language Hindi Named Entity Recognition task (Li and McCallum, 2003) had a similar setting. A name tagger was required to be finished within a time bound (five days). However, 628 labeled documents were provided in the TIDES task, while in our setting no labeled documents are available at the starting point. Therefore we applied active learning to efficiently annotate about 40 documents for each language and proposed new methods to learn expectations. The results of the tested ILs are still far from perfect, but we hope our detailed comparison and result analysis can introduce new ideas to balance the quality and cost of name tagging.

## 8 Conclusions and Future Work

Name tagging for a new IL is a very important but also challenging task. We conducted a thorough study on various ways of acquiring, encoding and composing expectations from multiple nontraditional sources. Experiments demonstrate that this framework can be used to build a promising name tagger for a new IL within a few hours. In the future we will exploit broader and deeper entity prior knowledge to improve name identification. We will aim to make the framework more transparent for native speakers so the survey can be done in an automatic interactive question-answering fashion. We will also develop methods to make the tagger capable of active self-assessment to produce the best workflow within time bounds.

## Acknowledgments

This work was supported by the U.S. DARPA LORELEI Program No. HR0011-15-C-0115 and ARL/ARO MURI W911NF-10-1-0533. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation here on.

## References

* Agichtein and Gravano [2000] Eugene Agichtein and Luis Gravano. 2000. Snowball: Extracting relations from large plain-text collections. In _Proceedings of the fifth ACM conference on Digital libraries_.
* Arevalo et al. [2002] Montse Arevalo, Xavier Carreras, Lluis Marquez, Maria Antonia Martí, Lluis Padro, and Maria Jose Simon. 2002. A proposal for wide-coverage spanish named entity recognition. _Procesamiento del lenguaje natural_.
* Asahara and Matsumoto [2003] Masayuki Asahara and Yuji Matsumoto. 2003. Japanese named entity extraction with redundant morphological analysis. In _Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology_.
* Banarescu et al. [2013] Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In _ACL Workshop on Linguistic Annotation and Interoperability with Discourse_.
* Bechet et al. [2000] Frederic Bechet, Alexis Nasr, and Franck Genet. 2000. Tagging unknown proper names using decision trees. In _Proceedings of the 38th Annual Meeting on Association for Computational Linguistics_.
* Becker et al. [2005] Markus Becker, Ben Hachey, Beatrice Alex, and Claire Grover. 2005. Optimising selective sampling for bootstrapping named entity recognition. In _Proceedings of ICML-2005 Workshop on Learning with Multiple Views_.
* Carreras et al. [2003] Xavier Carreras, Lluis Marquez, and Lluis Padro. 2003. Named entity recognition for catalan using spanish resources. In _Proceedings of the tenth conference on European chapter of the Association for Computational Linguistics_.
* Che et al. [2013] Wanxiang Che, Mengqiu Wang, Christopher D Manning, and Ting Liu. 2013. Named entity recognition with bilingual constraints. In _Proceedings of HLT-NACL_ 

Hai Leong Chieu and Hwee Tou Ng. 2002. Named entity recognition: a maximum entropy approach using global information. In _Proceedings of the 19th international conference on Computational linguistics_.
* Chiticariu et al. (2010) Laura Chiticariu, Rajasekar Krishnamurthy, Yunyao Li, Frederick Reiss, and Shivakumar Vaithyanathan. 2010. Domain adaptation of rule-based annotators for named-entity recognition tasks. In _Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing_.
* Collins and Singer (1999) Michael Collins and Yoram Singer. 1999. Unsupervised models for named entity classification. In _Proceedings of the joint SIGDAT conference on empirical methods in natural language processing and very large corpora_.
* Cucchiarelli et al. (1998) Alessandro Cucchiarelli, Danilo Luzi, and Paola Velardi. 1998. Automatic semantic tagging of unknown proper names. In _Proceedings of the 17th international conference on Computational linguistics_.
* Dalianis and Astrom (2001) Hercules Dalianis and Erik Astrom. 2001. Swenam--a swedish named entity recognizer. Technical report, Technical Report. Department of Numerical Analysis and Computing Science.
* De Meulder et al. (2002) Fien De Meulder, Walter Daelemans, and Veronique Hoste. 2002. A named entity recognition system for dutch. _Language and Computers_.
* Etzioni et al. (2005) Oren Etzioni, Michael Cafarella, Doug Downey, Ana-Maria Popescu, Tal Shaked, Stephen Soderland, Daniel S Weld, and Alexander Yates. 2005. Unsupervised named-entity extraction from the web: An experimental study. _Artificial intelligence_.
* Farmakiotou et al. (2000) Dimitra Farmakiotou, Vangelis Karkaletsis, John Koutsias, George Sigletos, Constantine D Spyropoulos, and Panagiotis Stamatopoulos. 2000. Rule-based named entity recognition for greek financial texts. In _Proceedings of the Workshop on Computational lexicography and Multimedia Dictionaries (COMLEX 2000)_.
* Finkel et al. (2005) Jenny Rose Finkel, Trond Grenager, and Christopher Manning. 2005. Incorporating non-local information into information extraction systems by gibbs sampling. In _Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics_.
* Harna et al. (2006) Jirka Harna, Anna Feldman, Chris Brew, and Luiz Amaral. 2006. Tagging portuguese with a spanish tagger using cognates. In _Proceedings of the International Workshop on Cross-Language Knowledge Induction_.
* Ji and Grishman (2005) Heng Ji and Ralph Grishman. 2005. Improving name tagging by reference resolution and relation detection. In _Proceedings of ACL2005_.
* Ji and Lin (2009) Heng Ji and Dekang Lin. 2009. Gender and animacy knowledge discovery from web-scale n-grams for unsupervised person mention detection. In _Proceedings of PACLIC2009_.
* Jing et al. (2004) Feng Jing, Mingjing Li, HongJiang Zhang, and Bo Zhang. 2004. Entropy-based active learning with support vector machines for content-based image retrieval. In _Proceedings of ICMCS2004_.
* Karkaletsis et al. (1999) Vangelis Karkaletsis, Georgios Paliouras, Georgios Petaasis, Natasa Manousopoulou, and Constantine D Spyropoulos. 1999. Named-entity recognition from greek and english texts. _Journal of Intelligent and Robotic Systems_.
* Kim et al. (2012) Sungchul Kim, Kristina Toutanova, and Hwanjo Yu. 2012. Multilingual named entity recognition using parallel data and metadata from wikipedia. In _Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics_.
* Lafferty et al. (2001) John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In _Proceedings of the Eighteenth International Conference on Machine Learning_.
* Li and McCallum (2003) Wei Li and Andrew McCallum. 2003. Rapid development of hindi named entity recognition using conditional random fields and feature induction. _ACM Transactions on Asian and Low-Resource Language Information Processing_.
* Li et al. (2012) Qi Li, Haibo Li, Heng Ji, Wen Wang, Jing Zheng, and Fei Huang. 2012. Joint bilingual name tagging for parallel corpora. In _Proceedings of the 21st ACM international conference on Information and knowledge management_.
* Li et al. (2014) Haibo Li, Masato Hagiwara, Qi Li, and Heng Ji. 2014. Comparison of the impact of word segmentation on name tagging for chinese and japanese. In _Proceedings of LREC2014_.
* Maloney and Niv (1998) John Maloney and Michael Niv. 1998. Tagarab: a fast, accurate arabic name recognizer using high-precision morphological analysis. In _Proceedings of the Workshop on Computational Approaches to Semitic Languages_.
* McCallum and Li (2003) Andrew McCallum and Wei Li. 2003. Early results for named entity recognition with conditional random fields, feature induction and web-enhanced lexicons. In _Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003_.
* Mikheev et al. (1999) Andrei Mikheev, Marc Moens, and Claire Grover. 1999. Named entity recognition without gazetteers. In _Proceedings of the ninth conference on European chapter of the Association for Computational Linguistics_.
* Mortimer and Salathiel (1995) JY Mortimer and JA Salathiel. 1995. 'soundex'codes of surnames provide confidentiality and accuracy in a national hiv database. _Communicable disease report. CDR review_.
* Nadeau and Sekine (2007) David Nadeau and Satoshi Sekine. 2007. A survey of named entity recognition and classification. _Lingvisti-cae Investigationes_ 

David Nadeau, Peter Turney, and Stan Matwin. 2006. Unsupervised named-entity recognition: Generating gazetteers and resolving ambiguity.
* [Nenadic and Spasic2000] Goran Nenadic and Irena Spasic. 2000. Recognition and acquisition of compound names from corpora. In _Natural Language Processing--NLP 2000_.
* [Niu et al.2003] Cheng Niu, Wei Li, Jihong Ding, and Rohini K Srihari. 2003. Bootstrapping for named entity tagging using concept-based seeds. In _Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology_.
* [Nothman et al.2013] Joel Nothman, Nicky Ringland, Will Radford, Tara Murphy, and James R Curran. 2013. Learning multilingual named entity recognition from wikipedia. _Artificial Intelligence_.
* [Och and Ney2003] Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. _Computational linguistics_.
* [Osenova and Kolkovska2002] Petya Osenova and Sia Kolkovska. 2002. Combining the named-entity recognition task and np chunking strategy for robust pre-processing. In _Proceedings of the Workshop on Treebanks and Linguistic Theories, September_.
* [Raghavan and Allan2004] Hema Raghavan and James Allan. 2004. Using soundex codes for indexing names in asr documents. In _Proceedings of the Workshop on Interdisciplinary Approaches to Speech Indexing and Retrieval at HLTNAACL 2004_.
* [Rizzo and Trrocy2012] Giuseppe Rizzo and Raphael Trrocy. 2012. Nerd: a framework for unifying named entity recognition and disambiguation extraction tools. In _Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics_.
* [Searle1980] John Searle. 1980. Minds, brains, and programs. _Journal of the Association for Computing Machinery_.
* [Settles and Craven2008] Burr Settles and Mark Craven. 2008. An analysis of active learning strategies for sequence labeling tasks. In _Proceedings of the conference on empirical methods in natural language processing_.
* [Settles2010] Burr Settles. 2010. Active learning literature survey. _University of Wisconsin, Madison_.
* [Thielen1995] Christine Thielen. 1995. An approach to proper name tagging for german. _arXiv preprint cmp-lg/9506024_.
* [Turn et al.2003] Gokhan Turn, Dilek Hakkani-Tur, and Kemal Oflazer. 2003. A statistical information extraction system for turkish. _Natural Language Engineering_.
* [Wang et al.2013] Mengqiu Wang, Wanxiang Che, and Christopher D Manning. 2013. Joint word alignment and bilingual named entity recognition using dual decomposition. In _Proceedings of the Association for Computational Linguistics_.
* [Wang et al.2015] Han Wang, Jin Guang Zheng, Xiaogang Ma, Peter Fox, and Heng Ji. 2015. Language and domain independent entity linking with quantified collective validation. In _Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP2015)_.
* [Wu et al.2009] Dan Wu, Wee Sun Lee, Nan Ye, and Hai Leong Chieu. 2009. Domain adaptive bootstrapping for named entity recognition. In _Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing_.
* [Zhou and Su2002] GuoDong Zhou and Jian Su. 2002. Named entity recognition using an hmm-based chunk tagger. In _Proceedings of the 40th Annual Meeting on Association for Computational Linguistics_.