Venue: NAACL 2019

# Cross-lingual Multi-Level Adversarial Transfer to Enhance

Low-Resource Name Tagging

Lifu Huang\({}^{1}\), Heng Ji\({}^{1}\), Jonathan May\({}^{2}\)

\({}^{1}\) Computer Science Department, Rensselaer Polytechnic Institute

{huangl7,jih}@rpi.edu

\({}^{2}\) Information Sciences Institute, University of Southern California

jonmay@isi.edu

###### Abstract

We focus on improving name tagging for low-resource languages using annotations from related languages. Previous studies either directly project annotations from a source language to a target language using cross-lingual representations or use a shared encoder in a multitask network to transfer knowledge. These approaches inevitably introduce noise to the target language annotation due to mismatched source-target sentence structures. To effectively transfer the resources, we develop a new neural architecture that leverages multi-level adversarial transfer: (1) word-level adversarial training, which projects source language words into the same semantic space as those of the target language without using any parallel corpora or bilingual gazetteers, and (2) sentence-level adversarial training, which yields language-agnostic sequential features. Our neural architecture outperforms previous approaches on CoNLL data sets. Moreover, on 10 low-resource languages, our approach achieves up to 16% absolute F-score gain over all high-performing baselines on cross-lingual transfer without using any target-language resources.1

Footnote 1: Our programs will be released at [https://github.com/wilburOne/AdversarialNameTagger](https://github.com/wilburOne/AdversarialNameTagger)

## 1 Introduction

Low-resource language name tagging is an important but challenging task. An effective solution is to perform cross-lingual transfer, by leveraging the annotations from high-resource languages. Most of these efforts achieve cross-lingual annotation projection based on bilingual parallel corpora combining with automatic word alignment Yarowsky et al. (2001); Wang et al. (2013); Fang and Cohn (2016); Ehrmann et al. (2011); Ni et al. (2017), bilingual gazetteers Feng et al. (2017); Zirikly and Hagiwara (2015), cross-lingual word embedding Fang and Cohn (2017); Wang et al. (2017); Huang et al. (2018), or cross-lingual Wikification Kim et al. (2012); Nothman et al. (2013); Tsai et al. (2016); Pan et al. (2017), but these resources are still only available for dozens of languages. Recent efforts on multi-task learning model each language as one single task while all the tasks share the same encoding layer Yang et al. (2016); Lin et al. (2018). These methods can transfer knowledge via the shared encoder without using bilingual resources. However, different languages usually have different underlying sequence structures, as shown in Figure 1. Without an explicit constraint, the encoder is not guaranteed to extract language-independent sequential features. Moreover, when the size of annotated resources is not balanced, the encoder is likely to be biased toward the resource-dominant language.

Considering these challenges, we develop a new neural architecture which can effectively transfer resources from source languages to improve target language name tagging. Our neural architecture is built upon a state-of-the-art sequence tagger: bi-directional long short-term memory as input to conditional random fields (Bi-LSTM-CRF) Lample et al. (2016); Huang et al. (2015); Ma and

Figure 1: Example of parallel sentences between English (ENG), Spanish (ESP) and Dutch (NED) from Europaral Parallel Corpus Koehn (2005). The information units with the same color and superscript are aligned.



Hovy, 2016), integrated with multi-level adversarial transfer: (1) word level adversarial transfer, similar to Conneau et al. (2017), applying a projection function on the source language and a discriminator to distinguish each word of the target language from that of the source language, resulting in a bilingual shared semantic space; (2) sentence-level adversarial transfer, where a discriminator is trained to distinguish each sentence of the target language from that of the source language,2 and a sequence encoder is applied to each sentence of both languages to prevent the discriminator from correctly predicting the source of each sentence, yielding language-agnostic sequential features. These features can better facilitate the resource transfer from the source language to the target language.

Footnote 2: For the name tagging task, ‘sequence’ always means ‘sentence.’

Our contributions are twofold: (1) without requiring any parallel corpora or bilingual gazetteers, the multi-level adversarial approach can efficiently transfer annotated resources from the source language to the target language and improve target language name tagging; (2) In addition to outperforming previous high-performing baselines on CoNLL data sets, we also evaluate cross-lingual name tagging on 10 low-resource languages and achieve up to 16% absolute F-score gain over all baselines when there is no annotated resource for the target language.

## 2 Approach

### Approach Overview

Figure 2 shows the overview of our neural architecture. It consists of three components: Cross-lingual word embedding learning with adversarial training:Given pre-trained monolingual word embeddings for a target language \(t\) and a source language \(s\), we first apply a mapping function to each word representation from \(s\), then feed both the projected source word representations and the target word representations to a word discriminator to predict the language of each word. If the discriminator cannot distinguish the language of \(t\) from the projection of \(s\), then we consider \(t\) and the projection of \(s\) to be in a shared space.

Language-agnostic sequential feature extraction:For each sentence of \(t\) and \(s\), we apply a sequence encoder to extract sequential features, and a Convolutional Neural Network (CNN) (Krizhevsky et al., 2012) based sequence discriminator to predict the language source of each sentence. The sequence encoder is trained to prevent the sequence discriminator from correctly predicting the language of each sentence, such that it finally extracts language-agnostic sequential features.

Language-independent name taggerThe language-agnostic sequential features from both \(t\) and \(s\) are further fed into a context encoder to better capture and refine contextual information and a conditional random field (CRF) (Lafferty et al., 2001) based name tagger.

Next we show the details of each component in our architecture.

### Word-level Adversarial Transfer

To better leverage the resources from the source language, our first step is to construct a shared se 

mantic space where the words from the source and target languages are semantically aligned. Without requiring any bilingual gazetteers, recent efforts (Zhang et al., 2017; Conneau et al., 2017; Chen and Cardie, 2018) explore unsupervised approaches to learn cross-lingual word embeddings and achieve comparable performance to supervised methods. Following these studies, we perform word-level adversarial training to automatically align word representations from \(s\) and \(t\).

Formally, assume we are given pre-trained monolingual word embeddings \({\bf V}_{t}=\{{\bf v}^{t}_{1},{\bf v}^{t}_{2},...,{\bf v}^{t}_{N}\}\in\mathbb{ R}^{N\times d_{t}}\) for \(t\), and \({\bf V}_{s}=\{{\bf v}^{s}_{1},{\bf v}^{s}_{2},...,{\bf v}^{s}_{M}\}\in\mathbb{R }^{M\times d_{s}}\) for \(s\), where \({\bf v}^{t}_{i}\) and \({\bf v}^{s}_{j}\) are the vector representations of words \(w^{t}_{i}\) and \(w^{s}_{i}\) from \(t\) and \(s\), \(N\) and \(M\) denote the vocabulary sizes, \(d_{t}\) and \(d_{s}\) denote the embedding dimensionality of \(t\) and \(s\) respectively. We then apply a mapping function \(f\) to project \(s\) into the same semantic space as \(t\):

\[\widetilde{\mathbf{V}}_{s}=f({\bf V}_{s})={\bf V}_{s}{\bf U} \tag{1}\]

where \({\bf U}\in\mathbb{R}^{d_{s}\times d_{t}}\) is the transformation matrix. \(\widetilde{\mathbf{V}}_{s}\in\mathbb{R}^{M\times d_{t}}\) are the projected word embeddings for \(s\), and \(\Theta_{f}=\{\theta_{f}\}\) denotes the set of parameters to be optimized for \(f\).

Similar to Xing et al. (2015), Conneau et al. (2017), and Chen and Cardie (2018), we constrain the transformation matrix \({\bf U}\) to be orthogonal with singular value decomposition (SVD) to reduce the parameter search space:

\[{\bf U}={\bf AB}^{\top}~{},\text{with}~{}~{}{\bf A}\Sigma{\bf B}^{\top}={\rm SVD }(\widetilde{\mathbf{V}}_{s}{\bf V}_{s}^{\top}) \tag{2}\]

To automatically optimize the mapping function \(f\) without using extra bilingual signals, we introduce a multi-layer perceptron \(D\) as a word discriminator, which takes word embeddings of \(t\) and projected word embeddings of \(s\) as input features and outputs a single scalar. \(D(w^{*}_{i})\) represents the probability of \(w^{*}_{i}\) coming from \(t\). The word discriminator is trained by minimizing the binary cross-entropy loss:

\[L^{w}_{dis}= -\frac{1}{I_{t;s}}\cdot\sum_{i=0}^{I_{t;s}}\Big{(}y_{i}\cdot \text{log}(D(w^{*}_{i}))\] \[+(1-y_{i})\cdot\text{log}(1-D(w^{*}_{i}))\Big{)}~{},\] \[y_{i}= \delta_{i}(1-2\epsilon)+\epsilon~{},\]

where \(\delta_{i}=1\) when \(w^{*}_{i}\) is from \(t\) and \(\delta_{i}=0\) otherwise. \(I_{t;s}\) represents the number of words sampled from the vocabulary of \(t\) and \(s\) together. \(\epsilon\) is a smoothed value added to the positive and negative labels. \(\Theta_{dis}=\{\theta_{D}\}\) is the parameter set.

The mapping function \(f\) and word discriminator \(D\) are two adversarial players, thus we flip the word labels and optimize \(f\) by minimizing the following loss:

\[L^{w}_{f}= -\frac{1}{I_{t;s}}\cdot\sum_{i=0}^{I_{t;s}}\left((1-y_{i})\cdot \text{log}(D(w^{*}_{i}))\right.\] \[+y_{i}\cdot\text{log}(1-D(w^{*}_{i}))\Big{)}~{},\]

\[y_{i}= \delta_{i}(1-2\epsilon)+\epsilon\]

Following the standard training procedures of deep adversarial networks (Goodfellow et al., 2014), we train the word discriminator and the mapping function successively with stochastic gradient descent (SGD) (Bottou, 2010) to minimize \(L^{w}_{dis}\) and \(L^{w}_{f}\). Similar to Conneau et al. (2017), after word-level adversarial training, we also adopt a refinement step to construct a bilingual dictionary for the top-\(k\) most frequent words in the source language3 based on \(\widetilde{\mathbf{V}}_{s}\) and \(\mathbf{V}_{t}\), and further optimize \({\bf U}\) with Equation 2 in a supervised way.

Footnote 3: We set k=15,000 in our experiment.

### Sentence-level Adversarial Transfer

Once \(s\) is projected into the same semantic space as \(t\), we can regard both sentences as coming from one unified language and directly project annotations from \(s\) to \(t\). However, name tagging not only relies on word level features, but also on sequential contextual features for entity type classification. Without constraints, the sequence encoder can only extract sequential features for both \(t\) and \(s\) based on their final training signals while these features are not necessarily beneficial to the target language. Thus, we further design sentence level adversarial transfer to encourage the encoder to extract language-agnostic sequential features.

Given a sentence \(x^{t}=\{w^{t}_{1},w^{t}_{2},...\}\) from \(t\) and a sentence \(x^{s}=\{w^{s}_{1},w^{s}_{2},...\}\) from \(s\), we first use \({\bf V}_{t}\) and \(\widehat{\mathbf{V}}_{s}\) to initialize a vector representation for each \(w^{t}_{i}\) and \(w^{s}_{i}\). We also apply a character-based CNN (denoted as CharCNN) (Kim et al., 2016) for each language to compose a word representation from its characters. For each word, we 

concatenate its word representation and character based representation. Then we feed the sequence of vector representations into a weight sharing Bi-LSTM encoder \(E\) to obtain sequential features \({\textbf{H}_{t}=\{{\bf h}^{t}_{1},{\bf h}^{t}_{2},...\}\) and \({\textbf{H}_{s}=\{{\bf h}^{s}_{1},{\bf h}^{s}_{2},...\}\) for \(x^{t}\) and \(x^{s}\) respectively. The parameter set of optimizing both language-dependent CharCNN and the sequence encoder can be denoted as \(\Theta_{e}=\{\theta_{\text{CharCNN}_{t}},\theta_{\text{CharCNN}_{s}},\theta_{ E}\}\).

Based on these sequential features, we use a sequence discriminator to predict the language source of each sentence. Given a sentence \(x^{*}\) and its sequential features \({\textbf{H}=\{{\bf h}^{*}_{1},{\bf h}^{*}_{2},...\}}\) from \(E\), we first apply a language-independent CNN with max-pooling to get an overall vector representation for \(x^{*}\), then feed it into another multi-layer perceptron, \(\tilde{D}\), to predict the probability that \(x^{*}\) comes from language \(t\). The sequence discriminator is trained by minimizing the following binary cross-entropy loss:

\[L^{x}_{dis}= -\frac{1}{\tilde{I}_{t;s}}\cdot\sum_{i=0}^{\tilde{I}_{t;s}}\left( \tilde{y}_{i}\cdot\textrm{log}(\tilde{D}(x^{*}_{i}))\right.\] \[+\left(1-\tilde{y}_{i}\right)\cdot\textrm{log}(1-\tilde{D}(x^{*} _{i}))\right),\]

\[\tilde{y}_{i}= \tilde{\delta}_{i}(1-2\eta)+\eta~{},\]

where \(\tilde{\delta}_{i}=1\) if the sentence \(x^{*}_{i}\) is from \(t\) and \(\tilde{\delta}_{i}=0\) otherwise. \(\tilde{I}_{t;s}\) represents the number of sentences sampled from the whole data set of \(t\) and \(s\). \(\eta\) is another smoothed value for sequence labels. \(\Theta_{\widetilde{dis}}=\{\theta_{\text{CNN}},\theta_{\tilde{D}}\}\) denotes the parameter set for optimizing the sequence discriminator.

The sequence encoder \(E\) and the sequence discriminator \(\tilde{D}\) are two adversarial players and \(E\) is optimized by trying to fool \(\tilde{D}\) to correctly predict the language source of each sentence. Thus we flip the sequence labels and optimize \(E\) by minimizing the following loss:

\[L^{x}_{e}= -\frac{1}{\tilde{I}_{t;s}}\cdot\sum_{i=0}^{\tilde{I}_{t;s}}\left( (1-\tilde{y}_{i})\cdot\textrm{log}(\tilde{D}(x^{*}_{i}))\right.\] \[+\left.\tilde{y}_{i}\cdot\textrm{log}(1-\tilde{D}(x^{*}_{i})) \right),\]

\[\tilde{y}_{i}= \tilde{\delta}_{i}(1-2\eta)+\eta\]

### Name Tagger Training

With the language-agnostic sequential features from \(E\), we can directly combine all annotated training data from both \(t\) and \(s\) to train the name tagger for \(t\). To do so, we feed the sequential features from \(E\) to another Bi-LSTM encoder \(E_{c}\) to refine the context information for each token, and use a CRF output layer to render predictions for each token, which can effectively capture dependencies among name tags (e.g., an "inside-organization" token cannot follow a "beginning-person" token).

Specifically, given an input sentence \(x=\{w_{1},w_{2},...w_{n}\}\), we extract language-agnostic sequential features with \(E\), and further obtain a new sequence of contextual features \(\mathbf{\tilde{H}=\{\tilde{h}_{1},\tilde{h}_{2},...,\tilde{h}_{n}\}\) with \(E_{c}\). Then we a apply a linear layer \(\ell\) to further convert each \(\tilde{\textbf{h}}_{i}\) to a score vector \(\textbf{y}_{i}\), in which each dimension denotes the predicted score for a tag (the starting, inside or outside of a name mention with a pre-defined entity type). Then we feed the sequence of score vectors \(\textbf{Y}=\{\textbf{y}_{1},\textbf{y}_{2},...,\textbf{y}_{n}\}\) into the CRF layer. The score of a sequence of tags \(\textbf{Z}=\{z_{1},z_{2},...,z_{n}\}\) is defined as:

\[Score(x,\textbf{Y},\textbf{Z})=\sum_{i=1}^{n}(R_{z_{i-1},z_{i}}+Y_{i,z_{i}})\]

where \(R\) is a transition matrix and \(R_{p,q}\) denotes the binary score of transitioning from tag \(p\) to tag \(q\) 

\(Y_{i,z}\) represents the unary score of assigning tag \(z\) to the \(i\)-th word.

Given the annotated sequence of tags **Z**, the CRF loss is:

\[L_{crf}=\text{log}\sum_{\textbf{Z}^{\prime}\in\hat{\mathbf{Z}}}e^{ Score(x,\textbf{Y},\textbf{Z}^{\prime})}-Score(x,\textbf{Y},\textbf{Z})\]

where \(\hat{\mathbf{Z}}\) is the set of all possible tagging paths. The parameter set for optimizing the name tagger can be denoted as \(\Theta_{c}=\{\theta_{E_{c}},\theta_{\ell},\theta_{\mathrm{CRF}}\}\).

We jointly optimize the sequence encoder \(E_{c}\), the context encoder \(E_{c}\) and the CRF together by minimizing the loss \(L^{{}^{\prime}}=L_{e}^{x}+L_{crf}\), and successively minimize \(L_{dis}^{x}\) and \(L^{{}^{\prime}}\) with SGD. The end-to-end training for our neural architecture is described in Algorithm 1.

## 3 Experiment

### Data and Experimental Setup

We evaluate our methods from multiple settings. We first evaluate our architecture on 10 low-resource languages from the DARPA LORELEI project. The annotations are released by the Linguistic Data Consortium (LDC).4 Each dataset has four predefined name types: person (PER), organization (ORG), location (LOC) and geo-political entity (GPE). For each target low-resource language, we choose a source language if they are from the same language family or use the same script. To show the impact of resource transfer between distinct languages, we also use English as a source language for each target low-resource language. We create the English annotated resource by combining the TAC-KBP 2015 English Entity Discovery and Linking Ji et al. (2015) data set and the Automatic Content Extraction (ACE2005) data set.5 To avoid the impact of parameter initialization, we perform 5-fold cross validation. For each experiment, we run twice and get the averaged F-score. Table 1 shows the statistics of each data set.

Footnote 4: The annotations are from: am (LDC2016E87), ti (LDC2017E39), ar (LDC2016E89), fa (LDC2016E93), om (LDC2017E27), so (LDC2016E91), sw (LDC2017E64), yo (LDC2016E105), ug (LDC2016E70), uz (LDC2016E29)

Footnote 5: The data sets are LDC2015E103 and LDC2006T06

We also evaluate our approach on high-resource languages. We use Dutch (nl) and Spanish (es) data sets from the CoNLL 2002 Tjong Kim Sang, 2002) shared task as target languages, and use English (en) data from the CoNLL 2003 Tjong Kim Sang and De Meulder (2003) shared task as the source language. All the data sets have four pre-defined name types: PER, ORG, LOC and miscellaneous (MISC). Table 2 shows the statistics of these data sets.

For fair comparison, we use the same pretrained word embeddings of English, Dutch and Spanish as Lin et al. (2018), while for each low-resource language we train their word embeddings using the documents from their LDC packages with FastText.6 Table 3 lists the key hyper-parameters we used in our experiments.

Footnote 6: [https://fasttext.cc/](https://fasttext.cc/)

### Baselines

We compare our methods with three categories of baseline methods:7

Footnote 7: All the baselines are trained for 100 epochs

* **Monolingual Name Tagging** Using monolingual annotations only, the current state-of-the-art name tagging model is the Bi-LSTM-CRF network Huang et al. (2015); Lample et al. (2016); Ma and Hovy (2016).8

Footnote 8: For each word, we also combine its word embedding with a CharCNN based representation.
* **Multi-task Learning**Lin et al. (2018) apply multi-task learning to boost name tagging performance by introducing additional annotations from source languages using a weight sharing context encoder across multiple languages.
* **Language Universal Representations** We apply word adversarial transfer only to project the source language into the same semantic space as the target language, then train the name tagger on the annotations of source and target languages. Word-Adv1 refers to the approach which is directly trained on the combination of the anno

\begin{table}
\begin{tabular}{l r r r} \hline Language & \# of Sents & \# of Tokens & \# of Names \\ \hline Amharic (am) & 4,770 & 71,399 & 3,891 \\ Tigriving (ti) & 5,023 & 95,364 & 6,201 \\ Arabic (ar) & 4,781 & 80,715 & 4,937 \\ Farsi (fa) & 3,855 & 72,629 & 3,966 \\ Oromo (om) & 2,987 & 52,876 & 4,985 \\ Somali (so) & 3,453 & 78,400 & 5,571 \\ Swahili (sw) & 4,155 & 96,902 & 6,044 \\ Yoruba (yo) & 1,599 & 46,084 & 2,016 \\ Uyghur (ug) & 3,961 & 60,999 & 2,575 \\ Uzbek (uz) & 11,135 & 177,816 & 10,937 \\ English (en) & 17,936 & 388,120 & 23,938 \\ \hline \end{tabular}
\end{table}
Table 1: Data set statistics for each low-resource language.

 

tations, while Word-Adv\({}^{2}\) refers to the baseline that is first trained on the target language annotations and then further tuned on the related language annotations.

### Cross-lingual Transfer with Zero Target Language Annotated Resource

We first evaluate our approach on a cross-lingual transfer setting without using any annotated training data from the target language. We conduct experiments on 8 low-resource languages. Among those, some pairs, such as Amharic (am) and Tigrinya (ti), Oromo (om) and Somali (so), or Yoruba (yo) and Swahili (sw), are from the same language family and are closely related, while some are not, such as Arabic (ar) and Farsi (fa). Since our approach requires some unlabeled sentences from the target language to train the sentence-level discriminator, we entirely remove the annotations from the annotated data set of the target language. Table 4 presents the results.

Our approach significantly outperforms the previous methods on all languages. Specifically, compared with the Word-Adv\({}^{1}\) baseline, which only performs word-level adversarial transfer, our approach achieves 10% absolute F-score gain on average, which demonstrates the effectiveness of the sentence-level adversarial transfer. In addition, compared with Lin et al. (2018), who only apply a shared context-encoder to transfer the knowledge, our approach not only includes a language-sharing encoder, but also performs multi-level adversarial training to encourage the semantic alignment of words from both languages and a sequence encoder to extract language-agnostic sequential features.

Here we use some Arabic (Farsi) examples to further show the effectiveness of each level of adversarial training in our architecture. Without using any annotated training data from Arabic, both our approach and the Word-Adv\({}^{1}\) baseline successfully identify \(\mathfrak{z}_{\text{embed}}\),\(\mathfrak{z}_{\text{embed}}\) (_French_) as a _GPE_ from the Arabic (ar) sentence in Figure 3, since with word-level adversarial training, the semantics of \(\mathfrak{z}_{\text{embed}}\),\(\mathfrak{z}_{\text{embed}}\) is well aligned with the _GPE_ names in Farsi annotated data, such as \(\mathfrak{z}_{\text{embed}}\),\(\mathfrak{z}_{\text{embed}}\) (_France_), \(\mathfrak{z}_{\text{embed}}\) (_Russia_) and \(\mathfrak{z}_{\text{embed}}\) (_Germany_). However, both the Word-Adv\({}^{1}\) and Lin et al. (2018) baselines fail to identify \(\mathfrak{z}_{\text{embed}}\),\(\mathfrak{z}_{\text{embed}}\) (_Algerian_) as a _GPE_ since its top ranked similar words in Farsi include \(\mathfrak{z}_{\text{embed}}\),\(\mathfrak{z}_{\text{embed}}\) (_negotiations_), \(\mathfrak{z}_{\text{embed}}\) (_Doha_) and \(\mathfrak{z}_{\text{embed}}\) (_agreement_). With sentence-level adversarial training, our approach successfully captures language-agnostic sequential features, such as "\(\mathfrak{gl}\) (_or_) _usually connects two names with the same type_", thus our approach successfully identifies \(\mathfrak{z}_{\text{embed}}\),\(\mathfrak{z}_{\text{embed}}\) (_Algerian_) as a _GPE_ name.

### Cross-lingual Transfer for Low-Resource Languages

We also investigate the impact of cross-lingual transfer when the target languages have some annotated resources. For each target low-resource language, we explore the use of a related low-resource language v

\begin{table}
\begin{tabular}{l r r r} \hline  & \multicolumn{1}{c}{Tran} & \multicolumn{1}{c}{Dev} & \multicolumn{1}{c}{Test} \\ \hline English (en) & source language & 204,567 (23,499) & 51,578 (5,942) & 46,666 (5,648) \\ Dutch (nl) & target language & 202,931 (13,344) & 37,761 (2,616) & 68,994 (3,941) \\ Spanish (es) & target language & 264,715 (18,797) & 52,923 (4,351) & 51,533 (3,558) \\ \hline \end{tabular}
\end{table}
Table 2: CoNLL data set statistics: # of tokens and # of names (between parentheses).

\begin{table}
\begin{tabular}{l c c} \hline Parameter Name & Value \\ \hline Monlongual Embedding Size & 100 \\ CharCNN Filter Size & 25 \\ CharCNN Filter Widths & [2, 3] \\ LSTM Hidden Size & 100 \\ Dropout Rate & 0.5 \\ Smoothing Value \(\epsilon\) for Word Discriminator & 0.1 \\ Word Adversarial Training Epochs & 5 \\ Smoothing Value \(\eta\) for Sequence Discriminator & 0.3 \\ Sequence Adversarial \& Name Tagging Training Epochs & 60 \\ \# of Steps for Sequence Tagging Training & 5 \\ Batch Size & 20 \\ Initial Learning Rate & 0.01 \\ Optimizer & SGD \\ \hline \end{tabular}
\end{table}
Table 3: Hyper-parameters.

 

glish as our source language. Table 5 shows the performance on 10 low-resource languages.

Comparing cross-lingual embedding based baselines to the monolingual baseline, we observe that for most low-resource languages, directly adding the annotations from the source language to the target language slightly hurts the model. This suggests that when the training data for the target language is not enough, the model will be very sensitive to noise. The multitask learning based baseline Lin et al. (2018) performs better than the monolingual baseline only when the target and source languages are very close, such as Amharic (am) and Tigrinya (ti), or Swahili (sw) and Yoruba (yo).

By introducing annotated training data from English, the performance of all the baselines becomes worse than the monolingual baseline. Since the script and sequence structure of English is very different from these low-resource languages, the addition of English to the limited target language training data yields a considerably noisy corpus. However, by forcing the sequence encoder to extract language-agnostic features, our approach still achieves better performance than the monolingual baseline for most languages. All of these experiments demonstrate that our approach is more effective in leveraging annotations from other languages to improve target language name tagging.

### Cross-lingual Transfer for High Resource Languages

We finally investigate the results when both the source and target languages ar

\begin{table}
\begin{tabular}{l l c} \hline Language & Model & F-score \\ \hline  & Lample et al. (2016) & 81.74 \\  & Yang et al. (2017) & 85.19 \\  & Lin et al. (2018) & 85.71 \\ Dutch & Gillick et al. (2016) & 82.84 \\  & Word-Adv\({}^{1}\) & 85.87 \\  & Word-Adv\({}^{2}\) & 86.43 \\  & Our Model (Bi-LSTM) & **86.87** \\  & Lample et al. (2016) & 85.75 \\  & Yang et al. (2017) & 85.77 \\  & Lin et al. (2018) & 85.02 \\ Spanish & GilICK et al. (2016) & 82.95 \\  & Word-Adv\({}^{1}\) & 85.92 \\  & Word-Adv\({}^{2}\) & 85.84 \\  & Our Model (Bi-LSTM) & **86.41** \\ \hline \end{tabular}
\end{table}
Table 6: Comparison on cross-lingual transfer for Dutch and Spanish with various baselines: monolingual baseline (Lample et al. (2016)), multitask baselines (Yang et al. (2017) and Lin et al. (2018)), language universal representation baselines (Gillick et al. (2016), Word-Adv\({}^{1}\), Word-Adv\({}^{2}\)).

Figure 3: Example of an Arabic (ar) name tagging output with Farsi (fa) annotated training data only.

 

languages. Table 6 presents the performance on Dutch and Spanish while using English as the source language. Our approach significantly outperforms all the other approaches even when the size of the annotated training data for the target language is huge. We notice that our approach achieves larger improvement on Dutch than Spanish. The reason may be that, compared with Spanish, Dutch is much closer to English Cutler and Pasveer (2006). Both English and Dutch are from the same _West Germanic_ branch of the _Indo-European_ language family while Spanish is from the _Italic_ branch.

### Impact of Annotation Size from Source and Target Languages

We use Amharic as the target language and Tigrinya as the source language to show the impact of the size of their annotations. Specifically, to explore the impact of the size of target language annotations, we use 0, 10%, 50%, or 100% annotated training data from Amharic. Similarly, to show the effect of the size of source language annotations, for each experiment, we also gradually add 0, 20%, 50%, or 100% annotated training data from Tigrinya. For all experiments, we use the same dev and test set of Amharic. As Figure 4 shows, as we gradually add annotations from the source or target language, the performance can always be improved. When the size of target language annotations is small, such as 400 sentences, we can achieve 5%-30% F-score gain by adding about 4,000 sentences from the source language. When the size of target language annotations is over 2,000 sentences, the improvement is about 2% if we add in about 4,000 sentences from source language annotations.

## 4 Related Work

Name tagging methods based on sequence labeling have been widely studied in recent years. Huang et al. (2015) and Lample et al. (2016) propose an effective Bi-LSTM-CRF architecture; the Bi-LSTM encodes previous and following contexts, and the CRF is used for tag prediction. Other studies incorporate a character-level CNN Ma and Hovy (2016), global contexts Zhang et al. (2018), or language models Liu et al. (2018); Peters et al. (2017, 2018); Devlin et al. (2018) to improve name tagging. In addition, several approaches Zhang et al. (2016a, 2017a); AlBadrashiny et al. (2017) attempt to incorporate hand-crafted linguistic features into a Bi-LSTM-CRF to improve low-resource name tagging performance.

Recent attempts on cross-lingual transfer for name tagging can be divided into two categories: the first projects annotations from a source language to a target language via parallel corpora Yarowsky et al. (2001); Wang and Manning (2013); Wang et al. (2013); Zhang et al. (2016b); Fang and Cohn (2016); Ehrmann et al. (2011); Enghoff et al. (2018); Ni et al. (2017), a bilingual gazetteer Feng et al. (2017); Ziriky and Hagiwara (2015), Wikipedia anchor links Kim et al. (2012); Nothman et al. (2013); Tsai et al. (2016); Pan et al. (2017), and language universal representations, including Unicode bytes Gillick et al. (2016) and cross-lingual word embeddings Fang and Cohn (2017); Wang et al. (2017); Huang et al. (2018); Xie et al. (2018). The second is based on multitask learning via a weight sharing encoder Yang et al. (2016, 2017); Lin et al. (2018). Compared to these studies, our approach not only automatically learns cross-lingual word embeddings without requiring any parallel resources, but also carefully extracts language-agnostic sequential features, yielding better performance.

Adversarial training has also been extensively studied and applied for cross-lingual and cross-domain transfer. Several studies Barone (2016); Zhang et al. (2017c, b); Conneau et al. (2017); Chen and Cardie (2018) explore adversarial training to automatically induce bilingual and multilingual word representations without using any parallel corpora or bilingual gazetteers. Adversarial training is also applied to extract language-agnostic Chen et al. (2016); Zou et al. (2018); Wang and Pan (2018); Kim et al. (2017a); Muis et al. (2019).

Figure 4: The impact of the size of annotations from source and target languages on Amharic name tagging.

 

2018; Cao et al., 2018) and domain-agnostic features Kim et al. (2017); Ganin et al. (2016); Tzeng et al. (2017); Chen et al. (2017); Li et al. (2017); Fu et al. (2017); Bousmalis et al. (2016); Shi et al. (2018) for cross-lingual and cross-domain adaptation. Compared with these methods, our approach combines both word-level and sentence-level adversarial training.

## 5 Conclusions and Future Work

We design a new neural architecture which integrates multi-level adversarial transfer into a BiLSTM-CRF to improve low-resource name tagging. With word-level adversarial training, it can automatically project the source language into a shared semantic space with the target language without requiring any comparable data or bilingual gazetteers. Moreover, considering the different underlying sequential structures among various languages, we further design a sentence-level adversarial transfer to encourage the sequence encoder to extract language-agnostic features. The experiments show that our approach achieves the state-of-the-art on both CoNLL data sets and 10 low-resource languages. In the future, we will further explore selecting the feature-consistent annotations from the source language and add to the target language, and explore unsupervised pretrained cross-lingual language models Peters et al. (2018); Radford et al. (2018); Devlin et al. (2018); Lample and Conneau (2019) for cross-lingual low resource name tagging.

## Acknowledgments

This research is based upon work supported in part by U.S. DARPA LORELEI Program \(\#\) HR0011-15-C-0115, and the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via contract \(\#\) FA8650-17-C-9116, and ARL NS-CTA No. W911NF-09-2-0053. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

## References

* Al-Badrashiny et al. (2017) Mohamed Al-Badrashiny, Jason Bolton, Arun Tejasvi Chaganty, Kevin Clark, Craig Harman, Lifu Huang, Matthew Lamm, Jinhao Lei, Di Lu, Xiaoman Pan, et al. 2017. Tinkerbell: Cross-lingual cold-start knowledge base construction. In _Proceedings of TAC 2017_.
* Barone (2016) Antonio Valerio Miceli Barone. 2016. Towards cross-lingual distributed representations without parallel text trained with adversarial autoencoders. _Proceedings of ACL 2016_.
* Bottou (2010) Leon Bottou. 2010. Large-scale machine learning with stochastic gradient descent. In _Proceedings of COMPSTAT 2010_.
* Bousmalis et al. (2016) Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. 2016. Domain separation networks. In _Proceedings of NIPS 2016_.
* Cao et al. (2018) Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao, and Shengping Liu. 2018. Adversarial transfer learning for chinese named entity recognition with self-attention mechanism. In _Proceedings of EMNLP 2018_, pages 182-192.
* Chen et al. (2017) Tseng-Hung Chen, Yuan-Hong Liao, Ching-Yao Chuang, Wan Ting Hsu, Jianlong Fu, and Min Sun. 2017. Show, adapt and tell: Adversarial training of cross-domain image captioner. In _Proceedings of ICCV 2017_.
* Chen and Cardie (2018) Xilun Chen and Claire Cardie. 2018. Unsupervised multilingual word embeddings. In _Proceedings of EMNLP 2018_.
* Chen et al. (2016) Xilun Chen, Yu Sun, Ben Athiwaratkun, Claire Cardie, and Kilian Weinberger. 2016. Adversarial deep averaging networks for cross-lingual sentiment classification. _arXiv preprint arXiv:1606.01614_.
* Conneau et al. (2017) Alexis Conneau, Guillaume Lample, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herve Jegou. 2017. Word translation without parallel data. _arXiv preprint arXiv:1710.04087_.
* Cutler and Pasever (2006) Anne Cutler and Dennis Pasever. 2006. Explaining cross-linguistic differences in effects of lexical stress on spoken-word recognition. In _3rd International Conference on Speech Prosody_. TUD press.
* Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. _arXiv preprint arXiv:1810.04805_.
* Ehrmann et al. (2011) Maud Ehrmann, Marco Turchi, and Ralf Steinberger. 2011. Building a multilingual named entity-annotated corpus using annotation projection. In _Proceedings of RANLP 2011_, pages 118-124.
* Enghoff et al. (2018) Jan Vium Enghoff, Soren Harrison, and Zeljko Agi. 2018. Low-resource named entity recognition via multi-source projection: Not quite there yet? In _The 4th Workshop on Noisy User-generated Text_ 

Meng Fang and Trevor Cohn. 2016. Learning when to trust distant supervision: An application to low-resource pos tagging using cross-lingual projection. _Proceedings of CoNLL 2016_.
* Fang and Cohn (2017) Meng Fang and Trevor Cohn. 2017. Model transfer for tagging low-resource languages using a bilingual dictionary. In _Proceedings of ACL 2017_.
* Feng et al. (2017) Xiaocheng Feng, Lifu Huang, Bing Qin, Ying Lin, Heng Ji, and Ting Liu. 2017. Multi-level cross-lingual attentive neural architecture for low resource name tagging. _Tsinghua Science and Technology_, pages 633-645.
* Fu et al. (2017) Lisheng Fu, Thien Huu Nguyen, Bonan Min, and Ralph Grishman. 2017. Domain adaptation for relation extraction with domain adversarial neural network. In _Proceedings of IJCNLP 2017_.
* Ganin et al. (2016) Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Francois Laviolette, Mario Marchand, and Victor Lempitsky. 2016. Domain-adversarial training of neural networks. _The Journal of Machine Learning Research_.
* Gillick et al. (2016) Dan Gillick, Cliff Brunk, Oriol Vinyals, and Amarna Subramanya. 2016. Multilingual language processing from bytes. In _Proceedings of NAACL-HLT 2016_.
* Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In _Proceedings of NIPS 2014_.
* Huang et al. (2018) Lifu Huang, Kyunghyun Cho, Boliang Zhang, Heng Ji, and Kevin Knight. 2018. Multi-lingual common semantic space construction via cluster-consistent word embedding. _Proceedings of EMNLP 2018_.
* Huang et al. (2015) Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging. _arXiv preprint arXiv:1508.01991_.
* Ji et al. (2015) Heng Ji, Joel Nothman, Ben Hachey, and Radu Florian. 2015. Overview of tac-kbp2015 tri-lingual entity discovery and linking. In _Proceedings of TAC 2015_.
* Kim et al. (2017a) Joo-Kyoung Kim, Young-Bum Kim, Ruhi Sarikaya, and Eric Fosler-Lussier. 2017a. Cross-lingual transfer learning for pos tagging without cross-lingual resources. In _Proceedings of EMNLP 2017_.
* Kim et al. (2012) Sunghul Kim, Kristina Toutanova, and Hwanjo Yu. 2012. Multilingual named entity recognition using parallel data and metadata from wikipedia. In _Proceedings of ACL 2012_, pages 694-702.
* Kim et al. (2017b) Taekoso Kim, Moonsu Cha, Hyunsso Kim, Jung Kwon Lee, and Jiwon Kim. 2017b. Learning to discover cross-domain relations with generative adversarial networks. In _Proceedings of ICML 2017_.
* Kim et al. (2016) Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. 2016. Character-aware neural language models. In _Proceedings of AAAI 2016_, pages 2741-2749.
* Koehn (2005) Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In _MT summit_, volume 5, pages 79-86.
* Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In _Proceedings of NIPS 2012_, pages 1097-1105.
* Lafferty et al. (2001) John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In _Proceedings of ICML 2001_.
* Lample et al. (2016) Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In _Proceedings of NAACL-HLT 2016_.
* Lample and Conneau (2019) Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. _arXiv preprint arXiv:1901.07291_.
* Li et al. (2017) Zheng Li, Yu Zhang, Ying Wei, Yuxiang Wu, and Qiang Yang. 2017. End-to-end adversarial memory network for cross-domain sentiment classification. In _Proceedings of IJCAI 2017_.
* Lin et al. (2018) Ying Lin, Shengqi Yang, Veselin Stoyanov, and Heng Ji. 2018. A multi-lingual multi-task architecture for low-resource sequence labeling. In _Proceedings of ACL 2018_, volume 1, pages 799-809.
* Liu et al. (2018) Liyuan Liu, Jingbo Shang, Xiang Ren, Frank Fangzheng Xu, Huan Gui, Jian Peng, and Jiawei Han. 2018. Empower sequence labelling with task-aware neural language model. In _Proceedings of AAAI 2018_.
* Ma and Hovy (2016) Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional lstm-cnns-crf. In _Proceedings of ACL 2016_.
* Muis et al. (2018) Aldrian Obaja Muis, Naoki Otani, Nidhi Vyas, Ruochen Xu, Yiming Yang, Teruko Mitamura, and Eduard Hovy. 2018. Low-resource cross-lingual event type detection via distant supervision with minimal effort. In _Proceedings COLING 2018_.
* Ni et al. (2017) Jian Ni, Georgiana Dinu, and Radu Florian. 2017. Weakly supervised cross-lingual named entity recognition via effective annotation and representation projection. In _Proceedings of ACL 2017_.
* Nothman et al. (2013)Joel Nothman, Nicky Ringland, Will Radford, Tara Murphy, and James R Curran. 2013. Learning multilingual named entity recognition from wikipedia. _Artificial Intelligence_, pages 151-175 

Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. Cross-lingual name tagging and linking for 282 languages. In _Proceedings of ACL 2017_, pages 1946-1958.
* Peters et al. (2017) Matthew Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. 2017. Semi-supervised sequence tagging with bidirectional language models. In _Proceedings of ACL 2017_.
* Peters et al. (2018) Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In _Proceedings of NAACL 2018_.
* Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training.
* Shi et al. (2018) Ge Shi, Chong Feng, Lifti Huang, Boliang Zhang, Heng Ji, Lejian Liao, and Heyan Huang. 2018. Genre separation network with adversarial training for cross-genre relation extraction. In _Proceedings of EMNLP 2018_.
* Sang (2002) Erik F. Tjong Kim Sang. 2002. Introduction to the conll-2002 shared task: Language-independent named entity recognition. In _Proceedings of CoNLL 2002_, pages 1-4.
* Sang and De Meulder (2003) Erik F T jong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In _Proceedings of HLT-NACL 2003_.
* Tsai et al. (2016) Chen-Tse Tsai, Stephen Mayhew, and Dan Roth. 2016. Cross-lingual named entity recognition via wikification. In _Proceedings of CoNLL 2016_.
* Tzeng et al. (2017) Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. 2017. Adversarial discriminative domain adaptation. In _Proceedings of CVPR 2017_.
* Wang et al. (2017) Dingquan Wang, Nanyun Peng, and Kevin Duh. 2017. A multi-task learning approach to adapting bilingual word embeddings for cross-lingual named entity recognition. In _Proceedings of IJCNLP 2017_.
* Wang et al. (2013) Mengqiu Wang, Wanxiang Che, and Christopher D Manning. 2013. Joint word alignment and bilingual named entity recognition using dual decomposition. In _Proceedings of ACL 2013_, pages 1073-1082.
* Wang and Manning (2013) Mengqiu Wang and Christopher D Manning. 2013. Cross-lingual pseudo-projected expectation regularization for weakly supervised learning. _arXiv preprint arXiv:1310.1597_.
* Wang and Pan (2018) Wenya Wang and Sinno Jialin Pan. 2018. Transition-based adversarial network for cross-lingual aspect extraction. In _Proceedings of IJCAI 2018_, pages 4475-4481.
* Xie et al. (2018) Jiateng Xie, Zhilin Yang, Graham Neubig, Noah A Smith, and Jaime Carbonell. 2018. Neural cross-lingual named entity recognition with minimal resources. In _Proceedings of EMNLP 2018_.
* Xing et al. (2015) Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. 2015. Normalized word embedding and orthogonal transform for bilingual word translation. In _Proceedings of NAACL 2015_.
* Yang et al. (2016) Zhilin Yang, Ruslan Salakhutdinov, and William Cohen. 2016. Multi-task cross-lingual sequence tagging from scratch. _arXiv preprint arXiv:1603.06270_.
* Yang et al. (2017) Zhilin Yang, Ruslan Salakhutdinov, and William W Cohen. 2017. Transfer learning for sequence tagging with hierarchical recurrent networks. _arXiv preprint arXiv:1703.06345_.
* Yarowsky et al. (2001) David Yarowsky, Grace Nagai, and Richard Wicentowski. 2001. Inducing multilingual text analysis tools via robust projection across aligned corpora. In _Proceedings of HLT 2001_.
* Zhang et al. (2017a)Boliang Zhang, Di Lu, Xiaoman Pan, Ying Lin, Haldanmu Abudukelimu, Heng Ji, and Kevin Knight. 2017a. Embracing non-traditional linguistic resources for low-resource language name tagging. In _Proceedings of IJCNLP 2017_, pages 362-372.
* Zhang et al. (2016a) Boliang Zhang, Xiaoman Pan, Tianlu Wang, Ashish Vaswani, Heng Ji, Kevin Knight, and Daniel Marcu. 2016a. Name tagging for low-resource incident languages based on expectation-driven learning. In _Proceedings of NAACL 2016_, pages 249-259.
* Zhang et al. (2018) Boliang Zhang, Spencer Whitehead, Lifa Huang, and Heng Ji. 2018. Global attention for name tagging. In _Proceedings of CoNLL 2018_, pages 86-96.
* Zhang et al. (2016b) Dongxu Zhang, Boliang Zhang, Xiaoman Pan, Xiaocheng Feng, Heng Ji, and XU Weiran. 2016b. Bi-text name tagging for cross-lingual entity annotation projection. In _Proceedings of COLING 2016_.
* Zhang et al. (2017b) Meng Zhang, Yang Liu, Huanbo Luan, and Maosong Sun. 2017b. Adversarial training for unsupervised bilingual lexicon induction. In _Proceedings of ACL 2017_.
* Zhang et al. (2017c) Meng Zhang, Yang Liu, Huanbo Luan, and Maosong Sun. 2017c. Earth mover's distance minimization for unsupervised bilingual lexicon induction. In _Proceedings of EMNLP 2017_.
* Zirikly and Hagiwara (2015) Ayah Zirikly and Masato Hagiwara. 2015. Cross-lingual transfer of named entity recognizers without parallel corpora. In _Proceedings of ACL 2015_.
* Zou et al. (2018) Bowei Zou, Zengzhuang Xu, Yu Hong, and Guodong Zhou. 2018. Adversarial feature adaptation for cross-lingual relation classification. In _Proceedings of COLING 2018_