Title: DATNET: DUAL ADVERSARIAL TRANSFER FOR LOW-RESOURCE NAMED ENTITY RECOGNITION

Abstract: We propose a new architecture termed Dual Adversarial Transfer Network (DAT-Net) for addressing low-resource Named Entity Recognition (NER). Specifically, two variants of DATNet, i.e., DATNet-F and DATNet-P, are proposed to explore effective feature fusion between high and low resource. To address the noisy and imbalanced training data, we propose a novel Generalized Resource-Adversarial Discriminator (GRAD). Additionally, adversarial training is adopted to boost model generalization. We examine the effects of different components in DATNet across domains and languages, and show that significant improvement can be obtained especially for low-resource data. Without augmenting any additional hand-crafted features, we achieve new state-of-the-art performances on CoNLL and Twitter NER-88.16% F1 for Spanish, 53.43% F1 for WNUT-2016, and 42.83% F1 for WNUT-2017 1 .

1. INTRODUCTION:

Named entity recognition (NER) is an important step in most natural language processing (NLP) applications. It detects not only the type of named entity, but also the entity boundaries, which requires deep understanding of the contextual semantics to disambiguate the different entity types of same tokens. To tackle this challenging problem, most early studies were based on hand-crafted rules, which suffered from limited performance in practice. Current methods are devoted to developing learning based algorithms, especially neural network based methods, and have been advancing the state-of-the-art consecutively (Collobert et al., 2011; Huang et al., 2015; Lample et al., 2016; Chiu & Nichols, 2016; Ma & Hovy, 2016) . These end-to-end models generalize well on new entities based on features automatically learned from the data. However, when the annotated corpora is small, especially in the low resource scenario (Zhang et al., 2016) , the performance of these methods degrades significantly since the hidden feature representations cannot be learned adequately.

Recently, more and more approaches have been proposed to address low-resource NER. Early works (Chen et al., 2010; Li et al., 2012) primarily assumed a large parallel corpus and focused on exploiting them to project information from high-to low-resource. Unfortunately, such a large parallel corpus may not be available for many low-resource languages. More recently, cross-resource word embedding (Fang & Cohn, 2017; Adams et al., 2017; Yang et al., 2017) was proposed to bridge the low and high resources and enable knowledge transfer. Although the aforementioned transferbased methods show promising performance in low-resource NER, there are two issues deserved to be further investigated on: 1) Representation Difference -they did not consider the representation difference across resources and enforced the feature representation to be shared across languages/domains; 2) Resource Data Imbalance -the training size of high-resource is usually much larger than that of low-resource. The existing methods neglect such difference in their models, resulting in poor generalization.

In this work, we present an approach termed Dual Adversarial Transfer Network (DATNet) to address the above issues in a unified framework for low-resource NER. Specifically, to handle the representation difference, we first investigate on two architectures of hidden layers (we use bidirectional long-short term memory (BiLSTM) model as hidden layer) for transfer. The first one is that all the units in hidden layers are common units shared across languages/domains. The second one is composed of both private and common units, where the private part preserves the independent language/domain information. Extensive experiments are conducted to show their advantages over each other in different situations. On top of common units, the adversarial discriminator (AD) loss is introduced to encourage the resource-agnostic representation so that the knowledge from high resource can be more compatible with low resource. To handle the resource data imbalance issue, we further propose a variant of the AD loss, termed Generalized Resource-Adversarial Discriminator (GRAD), to impose the resource weight during training so that low-resource and hard samples can be paid more attention to. In addition, we create adversarial samples to conduct the Adversarial Training (AT), further improving the generalization and alleviating over-fitting problem. We unify two kinds of adversarial learning, i.e., GRAD and AT, into one transfer learning model, termed Dual Adversarial Transfer Network (DATNet) , to achieve end-to-end training and obtain the state-of-the-art performance on a series of NER tasks-88.16% F1 for CoNLL-2002 Spanish, 53.43% and 42.83% F1 for WNUT-2016 and 2017 . Different from prior works, we do not use additional hand-crafted features and do not use cross-lingual word embeddings while addressing the cross-language tasks.

2. RELATED WORK:



Named Entity Recognition:

NER is typically framed as a sequence labeling task which aims at automatic detection of named entities (e.g., person, organization, location and etc.) from free text (Marrero et al., 2013) . The early works applied CRF, SVM, and perception models with handcrafted features (Ratinov & Roth, 2009; Passos et al., 2014; Luo et al., 2015) . With the advent of deep learning, research focus has been shifting towards deep neural networks (DNN), which requires little feature engineering and domain knowledge (Lample et al., 2016; Zukov Gregoric et al., 2018) . Collobert et al. (2011) proposed a feed-forward neural network with a fixed sized window for each word, which failed in considering useful relations between long-distance words. To overcome this limitation, Chiu & Nichols (2016) presented a bidirectional LSTM-CNNs architecture that automatically detects word-and character-level features. Ma & Hovy (2016) further extended it into bidirectional LSTM-CNNs-CRF architecture, where the CRF module was added to optimize the output label sequence. Liu et al. (2018) proposed task-aware neural language model termed LM-LSTM-CRF, where character-aware neural language models were incorporated to extract characterlevel embedding under a multi-task framework.

Transfer Learning for NER Transfer learning can be a powerful tool to low resource NER tasks. To bridge high and low resource, transfer learning methods for NER can be divided into two types: the parallel corpora based transfer and the shared representation based transfer. Early works mainly focused on exploiting parallel corpora to project information between the high-and low-resource language (Yarowsky et al., 2001; Chen et al., 2010; Li et al., 2012; Feng et al., 2018) . For example, Chen et al. (2010) and Feng et al. (2018) proposed to jointly identify and align bilingual named entities. On the other hand, the shared representation methods do not require the parallel correspondence (Rei & Søgaard, 2018) . For instance, Fang & Cohn (2017) proposed cross-lingual word embeddings to transfer knowledge across resources. Yang et al. (2017) presented a transfer learning approach based on a deep hierarchical recurrent neural network (RNN), where full/partial hidden features between source and target tasks are shared. Ni et al. (Ni & Florian, 2016; Ni et al., 2017) utilized the Wikipedia entity type mappings to improve low-resource NER. Al-Rfou' et al. (2015) built massive multilingual annotators with minimal human expertise by using language agnostic techniques. Mayhew et al. (2017) created a cross-language NER system, which works well for very minimal resources by translate annotated data of high-resource into low-resource. Cotterell & Duh (2017) proposed character-level neural CRFs to jointly train and predict low-and high-resource languages. Pan et al. (2017) proposes a large-scale cross-lingual named entity dataset which contains 282 languages for evaluation. In addition, multi-task learning (Yang et al., 2016; Luong et al., 2016; Rei, 2017; Aguilar et al., 2017; Hashimoto et al., 2017; Lin et al., 2018) shows that jointly training on multiple tasks/languages helps improve performance. Different from transfer learning methods, multi-task learning aims at improving the performance of all the resources instead of low resource only.

Adversarial Learning Adversarial learning originates from Generative Adversarial Nets (GAN) (Goodfellow et al., 2014) , which shows impressing results in computer vision. Recently, many papers have tried to apply adversarial learning to NLP tasks. Liu et al. (2017) presented an adversarial multi-task learning framework for text classification. Gui et al. (2017) applied the adversarial discriminator to POS tagging for Twitter. Kim et al. (2017) proposed a language discriminator to enable language-adversarial training for cross-language POS tagging. Apart from adversarial discriminator, adversarial training is another concept originally introduced by (Szegedy et al., 2014; Goodfellow et al., 2015) to improve the robustness of image classification model by injecting malicious perturbations into input images. Recently, Miyato et al. (2017) proposed a semi-supervised text classification method by applying adversarial training, where for the first time adversarial perturbations were added onto word embeddings. Yasunaga et al. (2018) applied adversarial training to POS tagging. Different from all these adversarial learning methods, our method integrates both the adversarial discriminator and adversarial training in an unified framework to enable end-to-end training.

3. DUAL ADVERSARIAL TRANSFER NETWORK (DATNET):

In this section, we introduce DATNet in more details. We first describe a base model for NER, and then discuss two proposed transfer architectures for DATNet. Figure 1 : The general architecture of proposed models.

3.1. BASIC ARCHITECTURE:

We follow state-of-the-art models for NER task (Huang et al., 2015; Lample et al., 2016; Chiu & Nichols, 2016; Ma & Hovy, 2016) , i.e., LSTM-CNNs-CRF based structure, to build the base model. It consists of the following pieces: character-level embedding, word-level embedding, BiLSTM for feature representation, and CRF as the decoder. The character-level embedding takes a sequence of characters in the word as atomic units input to derive the word representation that encodes the morphological information, such as root, prefix, and suffix. These character features are usually encoded by character-level CNN or BiLSTM, then concatenated with word-level embedding to form the final word vectors. On top of them, the network further incorporates the contextual information using BiLSTM to output new feature representations, which is subsequently fed into CRF layer to predict label sequence. Although both of the word-level layer and the character-level layer can be implemented using CNNs or RNNs, we use CNNs for extracting character-level and RNNs for extracting word-level representation. Fig. 1 (a) shows the the architecture of the base model.

3.2. DUAL ADVERSARIAL TRANSFER ARCHITECTURE:



3.2.1. CHARACTER-LEVEL ENCODER:

Previous works have shown that character features can boost sequence labeling performance by capturing morphological and semantic information (Lin et al., 2018) . For low-resource dataset to obtain high-quality word features, character features learned from other language/domain may provide crucial information for labeling, especially for rare and out-of-vocabulary words. Character-level encoder usually contains BiLSTM (Lample et al., 2016) and CNN (Chiu & Nichols, 2016; Ma & Hovy, 2016) approaches. In practice, Reimers & Gurevych (2017) observed that the difference between the two approaches is statistically insignificant in sequence labeling tasks, but character-level CNN is more efficient and has less parameters. Thus, we use character-level CNN and share character features between high-and low-resource tasks to enhance the representations of low-resource.

3.2.2. WORD-LEVEL ENCODER:

To learn a better word-level representation, we concatenate character-level features of each word with a latent word embedding as

w i = [w char i , w emb i ],

where the latent word embedding w emb i is initialized with pre-trained embeddings and fixed during training. One unique characteristic of NER is that the historical and future input for a given time step could be useful for label inference. To exploit such a characteristic, we use a bidirectional LSTM architecture (Hochreiter & Schmidhuber, 1997) to extract contextualized word-level features. In this way, we can gather the information from the past and future for a particular time frame t as follows,

- → h t = lstm( - → h t-1 , w t ), ← - h t = lstm( ← - h t+1 , w t ).

After the LSTM layer, the representation of a word is obtained by concatenating its left and right context representation as follows,

h t = [ - → h t , ← - h t ].

To consider the resource representation difference on word-level features, we introduce two kinds of transferable word-level encoder in our model, namely DATNet-Full Share (DATNet-F) and DATNet-Part Share (DATNet-P). In DATNet-F, all the BiLSTM units are shared by both resources while word embeddings for different resources are disparate. The illustrative figure is depicted in the Fig. 1(c ). Different from DATNet-F, the DATNet-P decomposes the BiLSTM units into the shared component and the resource-related one, which is shown in the Fig. 1 (b).

3.2.3. GENERALIZED RESOURCE-ADVERSARIAL DISCRIMINATOR:

In order to make the feature representation extracted from the source domain more compatible with those from the target domain, we encourage the outputs of the shared BiLSTM part to be resourceagnostic by constructing a resource-adversarial discriminator, which is inspired by the Language-Adversarial Discriminator proposed by Kim et al. (2017) . Unfortunately, previous works did not consider the imbalance of training size for two resources. Specifically, the target domain consists of very limited labeled training data, e.g., 10 sentences. In contrast, labeled training data in the source domain are much richer, e.g., 10k sentences. If such imbalance was not considered during training, the stochastic gradient descent (SGD) optimization would make the model more biased to high resource (Lin et al., 2017b) . To address this imbalance problem, we impose a weight α on two resources to balance their influences. However, in the experiment we also observe that the easily classified samples from high resource comprise the majority of the loss and dominate the gradient. To overcome this issue, we further propose Generalized Resource-Adversarial Discriminator (GRAD) to enable adaptive weights for each sample (note that the sample here means each sentence of resource), which focuses the model training on hard samples.

To compute the loss of GRAD, the output sequence of the shared BiLSTM is firstly encoded into a single vector via a self-attention module (Bahdanau et al., 2015) , and then projected into a scalar r via a linear transformation. The loss function of the resource classifier is formulated as:

EQUATION

where I i∈D S , I i∈D T are the identity functions to denote whether a sentence is from high resource (source) and low resource (target), respectively; α is a weighting factor to balance the loss contribution from high and low resource; the parameter (1 -r i ) γ (or r γ i ) controls the loss contribution from individual samples by measuring the discrepancy between prediction and true label (easy samples have smaller contribution); and γ scales the contrast of loss contribution from hard and easy samples. In practice, the value of γ does not need to be tuned much and usually set as 2 in our experiment. Intuitively, the weighting factors α and (1 -r i ) γ reduce the loss contribution from high resource and easy samples, respectively. Note that though the resource classifier is optimized to minimize the resource classification error, when the gradients originated from the resource classification loss are back-propagated to the other model parts than the resource classifier, they are negated for parameter updates so that these bottom layers are trained to be resource-agnostic.

3.2.4. LABEL DECODER:

The label decoder induces a probability distribution over sequences of labels, conditioned on the word-level encoder features. In this paper, we use a linear chain model based on the first-order Markov chain structure, termed the chain conditional random field (CRF) Lafferty et al. (2001) , as the decoder. In this decoder, there are two kinds of cliques: local cliques and transition cliques. Specifically, local cliques correspond to the individual elements in the sequence. And transition cliques, on the other hand, reflect the evolution of states between two neighboring elements at time t -1 and t and we define the transition distribution as θ. Formally, a linear-chain CRF can be written as p(y|h

1:T ) = 1 Z(h 1:T ) exp T t=2 θ yt-1,yt + T t=1 W yt h t ,

where Z(h 1:T ) is a normalization term and y is the sequence of predicted labels as follows: y = y 1:T . Model parameters are optimized to maximize this conditional log likelihood, which acts as the objective function of the model. We define the loss function for source and target resources as follows, S =i log p(y|h 1:T ), T = i log p(y|h 1:T ).

3.2.5. ADVERSARIAL TRAINING:

So far our model can be trained end-to-end with standard back-propagation by minimizing the following loss:

= GRAD + S + T (2)

Recent works have demonstrated that deep learning models are fragile to adversarial examples Goodfellow et al. (2015) . In computer vision, those adversarial examples can be constructed by changing a very small number of pixels, which are virtually indistinguishable to human perception (Pin-Yu et al., 2018) . Recently, adversarial samples are widely incorporated into training to improve the generalization and robustness of the model, which is so-called adversarial training (AT) (Miyato et al., 2017) . It emerges as a powerful regularization tool to stabilize training and prevent the model from being stuck in local minimum. In this paper, we explore AT in context of NER. To be specific, we prepare an adversarial sample by adding the original sample with a perturbation bounded by a small norm to maximize the loss function as follows:

EQUATION

)

where Θ is the current model parameters set. However, we cannot calculate the value of η exactly in general, because the exact optimization with respect to η is intractable in neural networks. Following the strategy in Goodfellow et al. (2015) , this value can be approximated by linearizing it as follows,

EQUATION

where can be determined on the validation set. In this way, adversarial examples are generated by adding small perturbations to the inputs in the direction that most significantly increases the loss function of the model. We find such η against the current model parameterized by Θ, at each training step, and construct an adversarial example by x adv = x + η x . Noted that we generate this adversarial example on the word and character embedding layer, respectively, as shown in the Fig. 1(b ) and 1(c). Then, the classifier is trained on the mixture of original and adversarial examples to improve the generalization. To this end, we augment the loss in Eqn. 2 and define the loss function for adversarial training as:

EQUATION

where (Θ; x), (Θ; x adv ) represents the loss from an original example and its adversarial counterpart, respectively. Note that we present the AT in a general form for the convenience of presentation. For different samples, the loss and parameters should correspond to their counterparts. For example, for the source data with word embedding w S , the loss for AT can be defined as follows, AT = (Θ; w S ) + (Θ; w S,adv ) with w S,adv = w S + η w S and = GRAD + S . Similarly, we can compute the perturbations η c for char-embedding and η w T for target word embedding.

4. EXPERIMENTS:



4.1. DATASETS:

In order to evaluate the performance of DATNet, we conduct the experiments on following widely used NER datasets: CoNLL-2003 English NER (Kim & De, 2003) , CoNLL-2002 Spanish & Dutch NER (Kim, 2002) , WNUT-2016 & 2017 English Twitter NER (Zeman, 2017) . The statistics of these datasets are described in Table 1 . We use the official split of training/validation/test sets. Since our goal is to study the effects of transferring knowledge from high-resource dataset to low-resource dataset, unlike previous works (Collobert et al., 2011; Chiu & Nichols, 2016; Yang et al., 2017) to append one-hot gazetteer features to the input of the CRF layer, and the works (Partalas et al., 2016; Limsopatham & Collier, 2016; Aguilar et al., 2017) to introduce orthographic feature as additional input for learning social media NER in tweets, we do not experiment with hand-crafted features and only consider words and characters embeddings as the inputs of our model. In addition to the CoNLL and WNUT datasets, we also experiment on the cross-language named entity dataset described in Pan et al. (2017) , which contains datasets for 282 languages, to evaluate our methods and investigate the transferability of different linguistic families and branches in both low-and high-resource scenarios. We choose 9 languages in our experiment, where Galician (gl), West Frisian (fy), Ukrainian (uk) and Marathi (mr) are target languages, the corresponding source languages are Spanish (es), Dutch (nl), Russian (ru) and Hindi (hi), and Arabic (ar) is also a source language, which is from different linguistic family. Following the setting in Cotterell & Duh (2017) , we also simulate the low-and high-resource scenarios by creating 100 and 10,000 sentences split for training target language datasets, respectively. Then we create 1,000 sentences split for validation and test, respectively. For source languages, we create 10,000 sentence split for training only. For high-resource scenario, we only conduct experiments on Galician (gl-high) and Ukrainian (uk-high).

The list of selected datasets are described in Table 2 . 

4.2. EXPERIMENTAL SETUP:

We use 50-dimensional publicly available pre-trained word embeddings for English, Spanish and Dutch languages of CoNLL and WNUT datasets in our experiments, which are trained by word2vec package2 on the corresponding Wikipedia articles (2017-12-20 dumps) (Lin et al., 2018) . For the named entity datasets selected from Pan et al. (2017) , we use 300-dimensional pre-trained word embeddings trained by fastText package3 on Wikipedia (Bojanowski et al., 2017) , and the 30dimensional randomly initialized character embeddings are used for all the datasets. We set the filter number as 20 for char-level CNN and the dimension of hidden states of the word-level LSTM as 200 for both base model and DATNet-F. For DATNet-P, we set 100 for source, share, and target LSTMs dimension, respectively. Parameters optimization is performed by Adam optimizer (Kingma & Ba, 2014) with gradient clipping of 5.0 and learning rate decay strategy. We set the initial learning rate of β 0 = 0.001 for all experiments. At each epoch t, learning rate β t is updated using

β t = β 0 /(1 + ρ × t)

, where ρ is decay rate with 0.05. To reduce over-fitting, we also apply Dropout (Srivastava et al., 2014) to the embedding layer and the output of the LSTM layer, respectively.

4.3. COMPARISON WITH STATE-OF-THE-ART RESULTS:

In this section, we compare our approach with state-of-the-art (SOTA) methods on CoNLL and WNUT benchmark datasets. In the experiment, we exploit all the source data (i.e., CoNLL-2003 English NER) and target data to improve performance of target tasks. The averaged results with standard deviation over 10 repetitive runs are summarized in Table 4 summarizes the results of our methods under different cross-language transfer settings as well as the comparison with Cotterell & Duh (2017) . In this experiment, we study the transferability between languages not only from same linguistic family and branch, but also from different linguistic families or branches. According to the results, DATNets outperform the transfer method of Cotterell & Duh (2017) for both low-and high-resource scenarios within the same linguistic family and branch (i.e., in-family in-branch) transfer case. We also observe that: 1) For the low-resource scenario, transfer learning is significantly helpful for improving the performance of target datasets within both same and different linguistic family or branch (i.e., in/cross-family in/cross-branch) transfer cases, while the improvements are more prominent under the in-family in-branch case. 2) For the high-resource scenario, say, when the target language data is sufficient, the improvements of transfer learning are not very distinct compared with that for low-resource scenario under in-family in-branch case. We also find that there is no effect by transferring knowledge from Arabic to Galician and Ukrainian. We suspect that it is caused by the great linguistic differences between source and target languages, since, for example, Arabic and Galician are from totally different linguistic families. 

4.4. TRANSFER LEARNING PERFORMANCE:

In this section, we investigate on improvements with transfer learning under multiple low-resource settings with partial target data. To simulate a low-resource setting, we randomly select subsets of target data with varying data ratio at 0.05, 0.1, 0.2, 0.4, 0.6, and 1.0. For example, 20,748 training tokens are sampled from the training set under a data ratio of r = 0.1 for the dataset CoNLL-2002 Spanish NER (Cf. Table 1 ). The results for cross-language and cross-domain transfer are shown in Fig. 2 (a) and 2(b), respectively, where we compare the results with each part of DATNet under various data ratios. From those figures, we have the following observations: 1) both adversarial training and adversarial discriminator in DATNet consistently contribute to the performance improvement; 2) the transfer learning component in the DATNet consistently improve over the base model results and the improvement margin is more substantial when the target data In the second experiment, we further investigate DATNet on the extremely low resource cases, e.g., the number of training target sentences is 10, 50, 100, 200, 500 and 1,000. The setting is quite challenging and fewer previous works have studied before. The results are summarized in Table 5 . We have two interesting observations5 : 1) DATNet-F outperforms DATNet-P on cross-language transfer when the target resource is extremely low, however, this situation is reversed when the target dataset size is large enough (here for this specific dataset, the threshold is 100 sentences); 2) DATNet-F is always superior to DATNet-P on cross-domain transfer. For the first observation, it is because DATNet-F with more shared hidden units is more efficient to transfer knowledge than DATNet-P when data size is extremely small. For the second observation, because cross-domain transfer are in the same language, more knowledge is common between the source and target domains, requiring more shared hidden features to carry with these knowledge compared to cross-language transfer. Therefore, for cross-language transfer with an extremely low resource and cross-domain transfer, we suggest using DATNet-F model to achieve better performance. As for cross-language transfer with relatively more training data, DATNet-P model is preferred. 

4.5. ABLATION STUDY OF DATNET:

In the proposed DATNet, both GRAD and AT play important roles in low resource NER. In this experiment, we further investigate how GRAD and AT help transfer knowledge across language/domain. In the first experiment6 , we used t-SNE (Maaten & Hinton, 2008) to visualize the feature distribution of BiLSTM outputs without AD, with normal AD (GRAD without considering data imbalance), and with the proposed GRAD in Figure 3 . From this figure, we can see that the GRAD in DATNet makes the distribution of extracted features from the source and target datasets much more similar by considering the data imbalance, which indicates that the outputs of BiLSTM are resource-invariant. To better understand the working mechanism, Table 6 further reports the quantitative performance comparison between models with different components. We observe that GRAD shows the stable superiority over the normal AD regardless of other components. There are no always winner between DATNet-P and DATNet-F on different settings. DATNet-P architecture is more suitable to cross-language transfer while DATNet-F is more suitable to cross-domain transfer. From the previous results, we know that AT helps enhance the overall performance by adding perturbations to inputs with the limit of = 5, i.e., η 2 ≤ 5. In this experiment, we further investigate how target perturbation w T with fixed source perturbation w S = 5 in AT affects knowledge transfer and the results on Spanish NER are summarized in Table 7 . The results generally indicate that less training data require a larger to prevent over-fitting, which further validates the necessity of AT in the case of low resource data. .37 78.63 78.70 78.32 77.96 77.92 77.88 77.78 77.85 77.90 77.65 77.57 77.38 77.49 77.29 ρ = 0.2 80.99 81.71 82.18 81.57 81.53 81.55 81.44 81.25 81.32 81.16 81.02 81.16 80.63 80.79 80.54 ρ = 0.4 83.76 83.73 84.18 84.48 84.26 84.12 83.54 83.40 83.52 84.18 83.42 83.47 83.28 83.33 83.19 ρ = 0.6 85.18 85.24 85.85 85.68 85.84 86.10 85.71 85.74 85.42 85.60 85.20 85.40 85.26 85.24 84.98 Finally, we analyze the discriminator weight α in GRAD and results are summarized in Table 8 . From the results, it is interesting to find that α is directly proportional to the data ratio ρ, basically, which means that more target training data requires larger α (i.e., smaller 1 -α to reduce training emphasis on the target domain) to achieve better performance.

5. CONCLUSION:

In this paper we develop a transfer learning model DATNet for low-resource NER, which aims at addressing two problems remained in existing work, namely representation difference and resource data imbalance. We introduce two variants of DATNet, DATNet-F and DATNet-P, which can be chosen for use according to the cross-language/domain user case and the target dataset size. To improve model generalization, we propose dual adversarial learning strategies, i.e., AT and GRAD. Extensive experiments show the superiority of DATNet over existing models and it achieves new state-of-the-art performance on CoNLL NER and WNUT NER benchmark datasets.

:

The implementation details will be available at https://github.com/ after acceptance.

https://github.com/tmikolov/word2vec

https://github.com/facebookresearch/fastText

Although our model performance on CONLL-2002 Dutch NER dataset is only comparable to the SOTA result, on the one hand, we do not use any addition features while the SOTA method did use; on the other, we are not sure if the SOTA method has incorporated the validation set into training. And if we merge training and validation sets, we can push the F1 score to 88.71, which outperforms the SOTA method.

For other tasks/languages we have the similar observation, we only reportCoNLL-2002 Spanish and WNUT-2016 Twitter results due to the page limit.

We used data ratio ρ = 0.5 for training model and randomly selected 10k testing data for visualization.

Figures and Tables:

Figure 2: Comparison with Different Target Data Ratio, where AT stands for adversarial training, F(P)-Transfer denotes the DATNet-F(P) without AT.

Figure 3: The visualization of extracted features from shared bidirectional-LSTM layer. The left, middle, and figures show the results when no Adversarial Discriminator (AD), AD, and GRAD is performed, respectively. Red points correspond to the source CoNLL-2003 English examples, and blue points correspond to the target CoNLL-2002 Spanish examples.



Table content: <table><tr><td>+</td></tr><tr><td>(b) DATNet-P</td></tr></table>



Table content: <table><tr><td>+</td></tr><tr><td>+</td></tr><tr><td>(c) DATNet-F</td></tr></table>

To be noted, we used only train set for model training for all datasets except the WNUT-2016 NER dataset. Since in this dataset, all the previous studies merged the training and validation sets together for training, we followed the same way for fair comparison. Specifically, we use CoNLL-2003 English NER dataset as high-resource (i.e., source) for all the experiments on CoNLL and WNUT datasets, whileCoNLL-2002 Spanish & Dutch NER datasets and WNUT-2016 & 2017  Twitter NER datasets as low-resource (i.e., target) in cross-language and cross-domain NER settings, respectively. Statistics of CoNLL and WNUT Named Entity Recognition Datasets.

Table content: <table><tr><td>Benchmark</td><td>Resource</td><td>Language</td><td># Training Tokens (# Entities)</td><td># Dev Tokens (# Entities)</td><td># Test Tokens (# Entities)</td></tr></table>

List of Named Entity Recognition Datasets inPan et al. (2017).

Table content: <table><tr><td>Language</td><td colspan="6">Resource Linguistic Family Linguistic Branch # Training Sentences # Dev Sentences # Test Sentences</td></tr><tr><td>Spanish (es)</td><td>Source</td><td>Indo-European</td><td>Romance</td><td>10,000</td><td>-</td><td>-</td></tr><tr><td>Galician (gl / gl-high)</td><td>Target</td><td>Indo-European</td><td>Romance</td><td>100 / 10,000</td><td>1,000</td><td>1,000</td></tr><tr><td>Dutch (nl)</td><td>Source</td><td>Indo-European</td><td>Germanic</td><td>10,000</td><td>-</td><td>-</td></tr><tr><td>West Frisian (fy)</td><td>Target</td><td>Indo-European</td><td>Germanic</td><td>100</td><td>1,000</td><td>1,000</td></tr><tr><td>Russian (ru)</td><td>Source</td><td>Indo-European</td><td>Slavic</td><td>10,000</td><td>-</td><td>-</td></tr><tr><td>Ukrainian (uk / uk-high)</td><td>Target</td><td>Indo-European</td><td>Slavic</td><td>100 / 10,000</td><td>1,000</td><td>1,000</td></tr><tr><td>Hindi (hi)</td><td>Source</td><td>Indo-European</td><td>Indo-Aryan</td><td>10,000</td><td>-</td><td>-</td></tr><tr><td>Marathi (mr)</td><td>Target</td><td>Indo-European</td><td>Indo-Aryan</td><td>100</td><td>1,000</td><td>1,000</td></tr><tr><td>Arabic (ar)</td><td>Source</td><td>Afro-Asiatic</td><td>Semitic</td><td>10,000</td><td>-</td><td>-</td></tr></table>

Table 3, and we also report the best results on each task for fair comparison with other SOTA methods. From results, we observe that incorporating the additional resource is helpful to improve performance. DATNet-P model achieves the highest F1 score on CoNLL-2002 Spanish and second F1 score on CoNLL-2002 Dutch dataset while DATNet-F model beats others on WNUT-2016 and WNUT-2017 English Twitter datasets. Different from other state-of-the-art models, DATNets do not use any addition features 4 .

Table content: <table/>

Comparison with State-of-the-art Results in CoNLL and WNUT datasets (F1-score).

Table content: <table><tr><td>Mode</td><td>Methods</td><td/><td colspan="3">Additional Features POS Gazetteers Orthographic</td><td colspan="2">CoNLL Datasets Spanish Dutch</td><td colspan="2">WNUT Datasets WNUT-2016 WNUT-2017</td></tr><tr><td>Mono-language /domain</td><td colspan="2">Gillick et al. (2016) Lample et al. (2016) Partalas et al. (2016) Limsopatham &amp; Collier (2016) Lin et al. (2017a)</td><td>× × √ × √</td><td>× √ √ × √</td><td>× × √ √ ×</td><td>82.59 85.75 ---</td><td>82.84 81.74 ---</td><td>--46.16 52.41 -</td><td>----40.42</td></tr><tr><td/><td>Our Base Model</td><td>Best Mean &amp; Std</td><td>×</td><td>×</td><td>×</td><td>85.53 85.35±0.15</td><td>85.55 85.24±0.21</td><td>44.96 44.37±0.31</td><td>35.20 34.67±0.34</td></tr><tr><td>Cross-language /domain</td><td colspan="3">Yang et al. (2017) Lin et al. (2018) Feng et al. (2018) von Däniken &amp; Cieliebak (2017) × × × √ Aguilar et al. (2017) √ DATNet-P Best Mean &amp; Std ×</td><td>√ √ × √ × ×</td><td>× × × × √ ×</td><td>85.77 85.88 86.42 --88.16 87.89±0.18</td><td>85.19 86.55 88.39 --88.32 88.09±0.13</td><td>-----50.85 50.41±0.32</td><td>---40.78 41.86 41.12 40.52±0.38</td></tr><tr><td/><td>DATNet-F</td><td>Best Mean &amp; Std</td><td>×</td><td>×</td><td>×</td><td>87.04 86.79±0.20</td><td>87.77 87.52±0.19</td><td>53.43 53.03±0.24</td><td>42.83 42.32±0.32</td></tr></table>

Results of Varying Cross-language Transfer Settings inPan et al. (2017) Datasets (F1-score). Base model means the model is trained by using target language dataset only.

Table content: <table><tr><td colspan="4">Language Source Target</td><td/><td colspan="2">Transferring Strategy</td><td colspan="4">Cotterell &amp; Duh (2017) Base Model Transfer</td><td colspan="2">Base Model</td><td>Our Methods DATNet-P</td><td>DATNet-F</td></tr><tr><td colspan="2">nl</td><td/><td>fy</td><td/><td colspan="2">In-Family In-Branch</td><td>58.43</td><td colspan="3">72.12</td><td colspan="2">57.47</td><td>75.08</td><td>76.05</td></tr><tr><td colspan="2">hi</td><td/><td>fy</td><td/><td colspan="2">In-Family Cross-Branch</td><td>-</td><td/><td>-</td><td/><td colspan="2">57.47</td><td>69.25</td><td>68.44</td></tr><tr><td colspan="2">ar</td><td/><td>fy</td><td/><td colspan="2">Cross-Family Cross-Branch</td><td>-</td><td/><td>-</td><td/><td colspan="2">57.47</td><td>67.89</td><td>66.05</td></tr><tr><td colspan="2">hi</td><td/><td>mr</td><td/><td colspan="2">In-Family In-Branch</td><td>39.02</td><td colspan="3">60.92</td><td colspan="2">43.55</td><td>68.55</td><td>64.87</td></tr><tr><td colspan="2">nl</td><td/><td>mr</td><td/><td colspan="2">In-Family Cross-Branch</td><td>-</td><td/><td>-</td><td/><td colspan="2">43.55</td><td>63.83</td><td>60.50</td></tr><tr><td colspan="2">ar</td><td/><td>mr</td><td/><td colspan="2">Cross-Family Cross-Branch</td><td>-</td><td/><td>-</td><td/><td colspan="2">43.55</td><td>63.28</td><td>59.76</td></tr><tr><td colspan="2">es</td><td/><td>gl</td><td/><td colspan="2">In-Family In-Branch</td><td>49.19</td><td colspan="3">76.40</td><td colspan="2">49.94</td><td>79.60</td><td>86.01</td></tr><tr><td colspan="2">hi</td><td/><td>gl</td><td/><td colspan="2">In-Family Cross-Branch</td><td>-</td><td/><td>-</td><td/><td colspan="2">49.94</td><td>60.57</td><td>61.68</td></tr><tr><td colspan="2">ar</td><td/><td>gl</td><td/><td colspan="2">Cross-Family Cross-Branch</td><td>-</td><td/><td>-</td><td/><td colspan="2">49.94</td><td>59.18</td><td>60.43</td></tr><tr><td colspan="2">es</td><td/><td colspan="2">gl-high</td><td colspan="2">In-Family In-Branch</td><td>89.42</td><td colspan="3">89.46</td><td colspan="2">92.78</td><td>93.14</td><td>93.02</td></tr><tr><td colspan="2">ar</td><td/><td colspan="2">gl-high</td><td colspan="2">Cross-Family Cross-Branch</td><td>-</td><td/><td>-</td><td/><td colspan="2">92.78</td><td>92.63</td><td>92.21</td></tr><tr><td colspan="2">ru</td><td/><td>uk</td><td/><td colspan="2">In-Family In-Branch</td><td>60.65</td><td colspan="3">76.74</td><td colspan="2">61.48</td><td>79.02</td><td>80.76</td></tr><tr><td colspan="2">hi</td><td/><td>uk</td><td/><td colspan="2">In-Family Cross-Branch</td><td>-</td><td/><td>-</td><td/><td colspan="2">61.48</td><td>72.73</td><td>73.84</td></tr><tr><td colspan="2">ar</td><td/><td>uk</td><td/><td colspan="2">Cross-Family Cross-Branch</td><td>-</td><td/><td>-</td><td/><td colspan="2">61.48</td><td>71.55</td><td>72.24</td></tr><tr><td colspan="2">ru</td><td/><td colspan="2">uk-high</td><td colspan="2">In-Family In-Branch</td><td>87.39</td><td colspan="3">87.42</td><td colspan="2">93.29</td><td>93.62</td><td>93.51</td></tr><tr><td colspan="2">ar</td><td/><td colspan="2">uk-high</td><td colspan="2">Cross-Family Cross-Branch</td><td>-</td><td/><td>-</td><td/><td colspan="2">93.29</td><td>92.83</td><td>92.42</td></tr><tr><td>F1 Score (%)</td><td>68 70 72 74 76 78 80 82 84 86 88</td><td>0.05</td><td>0.1</td><td colspan="2">0.2 Target Dataset Ratio 0.4</td><td>0.6 Base Base + AT 1.0 F-Transfer P-Transfer DATNet-F DATNet-P</td><td/><td>F1 Score (%)</td><td>16 18 26 30 32 34 36 38 52 50 48 46 44 42 40 28 24 22 20</td><td>0.05</td><td>0.1</td><td>0.2 Target Dataset Ratio 0.4</td><td>0.6 Base Base + AT 1.0 F-Transfer P-Transfer DATNet-F DATNet-P</td></tr><tr><td/><td/><td colspan="5">(a) CoNLL-2002 Spanish NER</td><td/><td/><td/><td colspan="3">(b) WNUT-2016 Twitter NER</td></tr></table>

Experiments on Extremely Low Resource (F1-score).

Table content: <table><tr><td>Tasks</td><td/><td colspan="4">CoNLL-2002 Spanish NER</td><td/><td/><td colspan="4">WNUT-2016 Twitter NER</td><td/></tr><tr><td># Target train sentences</td><td>10</td><td>50</td><td>100</td><td>200</td><td>500</td><td>1000</td><td>10</td><td>50</td><td>100</td><td>200</td><td>500</td><td>1000</td></tr><tr><td>Base</td><td>21.53</td><td>42.18</td><td>48.35</td><td>63.66</td><td>68.83</td><td>76.69</td><td>3.80</td><td>14.07</td><td>17.99</td><td>26.20</td><td>31.78</td><td>36.99</td></tr><tr><td>+ AT</td><td>19.23</td><td>41.01</td><td>50.46</td><td>64.83</td><td>70.85</td><td>77.91</td><td>4.34</td><td>16.87</td><td>18.43</td><td>26.32</td><td>35.68</td><td>41.69</td></tr><tr><td>+ P-Transfer</td><td>29.78</td><td>61.09</td><td>64.78</td><td>66.54</td><td>72.94</td><td>78.49</td><td>7.71</td><td>16.17</td><td>20.43</td><td>29.20</td><td>34.90</td><td>41.20</td></tr><tr><td>+ F-Transfer</td><td>39.72</td><td>63.00</td><td>63.36</td><td>66.39</td><td>72.88</td><td>78.04</td><td>15.26</td><td>20.04</td><td>26.60</td><td>32.22</td><td>38.35</td><td>44.81</td></tr><tr><td>DATNet-P</td><td>39.52</td><td>62.57</td><td>64.05</td><td>68.95</td><td>75.19</td><td>79.46</td><td>9.94</td><td>17.09</td><td>25.39</td><td>30.71</td><td>36.05</td><td>42.30</td></tr><tr><td>DATNet-F</td><td>44.52</td><td>63.89</td><td>66.67</td><td>68.35</td><td>74.24</td><td>78.56</td><td>17.14</td><td>22.59</td><td>28.41</td><td>32.48</td><td>39.20</td><td>45.25</td></tr><tr><td colspan="13">ratio is lower. For example, when the data ratio is 0.05, DATNet-P model outperforms the base</td></tr><tr><td colspan="13">model by more than 4% absolutely in F1-score on Spanish NER and DATNet-F model improves</td></tr><tr><td colspan="11">around 13% absolutely in F1-score compared to base model on WNUT-2016 NER.</td><td/><td/></tr></table>

Quantitative Performance Comparison between Models with Different Components. AT: Adversarial Training; P-T: P-Transfer; F-T: F-Transfer; AD: Adversarial Discriminator; GRAD: Generalized Resource-Adversarial Discriminator.

Table content: <table><tr><td/><td colspan="2">CoNLL-2002 Spanish NER</td><td/><td/><td colspan="2">WNUT-2016 Twitter NER</td><td/></tr><tr><td>Model</td><td colspan="2">F1-score Model</td><td colspan="2">F1-score Model</td><td colspan="2">F1-score Model</td><td>F1-score</td></tr><tr><td>Base</td><td>85.35</td><td>+AT</td><td>86.12</td><td>Base</td><td>44.37</td><td>+AT</td><td>47.41</td></tr><tr><td>+P-T (no AD)</td><td>86.15</td><td>+AT +P-T (no AD)</td><td>86.90</td><td>+P-T (no AD)</td><td>47.66</td><td>+AT +P-T (no AD)</td><td>48.44</td></tr><tr><td>+F-T (no AD)</td><td>85.46</td><td>+AT +F-T (no AD)</td><td>86.17</td><td>+F-T (no AD)</td><td>49.79</td><td>+AT +F-T (no AD)</td><td>50.93</td></tr><tr><td>+P-T (AD)</td><td>86.32</td><td>+AT +P-T (AD)</td><td>87.19</td><td>+P-T (AD)</td><td>48.14</td><td>+AT +P-T (AD)</td><td>49.41</td></tr><tr><td>+F-T (AD)</td><td>85.58</td><td>+AT +F-T (AD)</td><td>86.38</td><td>+F-T (AD)</td><td>50.48</td><td>+AT +F-T (AD)</td><td>51.84</td></tr><tr><td>+P-T (GRAD)</td><td>86.93</td><td>+AT +P-T (GRAD) (DATNet-P)</td><td>88.16</td><td>+P-T (GRAD)</td><td>48.91</td><td>+AT +P-T (GRAD) (DATNet-P)</td><td>50.85</td></tr><tr><td>+F-T (GRAD)</td><td>85.91</td><td>+AT +F-T (GRAD) (DATNet-F)</td><td>87.04</td><td>+F-T (GRAD)</td><td>51.31</td><td>+AT +F-T (GRAD) (DATNet-F)</td><td>53.43</td></tr></table>

Analysis of Maximum Perturbation w T in AT with Varying Data Ratio ρ (F1-score).

Table content: <table><tr><td>w T</td><td>1.0</td><td>3.0</td><td>5.0</td><td>7.0</td><td>9.0</td></tr><tr><td>Ratio</td><td/><td colspan="3">CoNLL-2002 Spanish NER</td><td/></tr><tr><td>ρ = 0.1</td><td>75.90</td><td>76.23</td><td>77.38</td><td>77.77</td><td>78.13</td></tr><tr><td>ρ = 0.2</td><td>81.54</td><td>81.65</td><td>81.32</td><td>81.81</td><td>81.68</td></tr><tr><td>ρ = 0.4</td><td>83.62</td><td>83.83</td><td>83.43</td><td>83.99</td><td>83.40</td></tr><tr><td>ρ = 0.6</td><td>84.44</td><td>84.47</td><td>84.72</td><td>84.04</td><td>84.05</td></tr></table>

Analysis of Discriminator Weight α in GRAD with Varying Data Ratio ρ (F1-score).

Table content: <table><tr><td>α</td><td>0.1</td><td>0.15</td><td>0.2</td><td>0.25</td><td>0.3</td><td>0.35</td><td>0.4</td><td>0.45</td><td>0.5</td><td>0.55</td><td>0.6</td><td>0.65</td><td>0.7</td><td>0.75</td><td>0.8</td></tr><tr><td>Ratio</td><td/><td/><td/><td/><td/><td/><td colspan="3">CoNLL-2002 Spanish NER</td><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="2">ρ = 0.1 78</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr></table>

Peer review (iclr2019):
 <Summary> Authors propose the new “DATNet” for the NER task, which extends the base neural model for NER (Bi-LSTM+CRF sequence model with input represented with CharCNN-word embeddings) with the following two main components: (1) GRAD: a language (or resource) adversarial discriminator with adaptive weights that regularize source-target data imbalance, and (2) additional adversarial training approaches that perturb input samples in the embeddings space. The paper reports big improvement over their baseline approaches without having to rely on other auxiliary or hand-crafted features. The experiment is performed for various low-resource scenarios (varying training data size). <Comments> - While the idea of applying “dual adversarial” approaches is new in the context of NER, the technical novelty of each component is limited. GRAD, for example, is rather a minor modification of Language Adversarial Discriminator (Kim et al., 2017) with a scalar weight parameter on loss. The empirical superiority of the proposed method (GRAD) over normal AD approaches cannot be claimed either -- for this authors need to report quantitative results of the (Base + AT + F/P-transfer with AD -- or e.g. .alpha fixed at 0.5 or at some other rate), which I believe is missing in all figures and tables. Visualization of resulting feature distribution (Figure 3) is interesting to look at, but that alone does not suffice. There is no technical novelty in applying adversarial training either, except that it was used in the context of NER. - Authors use both source and target data to train their “base model” as well (“... we exploit all the source data and target data ...“), e.g. presumably by merging the source and target dataset as well as their embedding matrices, etc. It is important that authors report if there ever is a negative transfer case (e.g. a base model trained with just target data may outperform models trained with source+target data) at varying resource scenarios -- especially at sufficiently-resourced cases. If by any chance their “base model” refers to in-domain training / in-domain testing results on target, the aforementioned baseline (naive merge of source and target data) is obviously an important baseline to report. I suggest that authors provide these details or clarify. (The confusion comes mainly because some of the SOTA results the authors quote are in-domain training / in-domain evaluation results on respective languages, and some are cross-domain results -- yet they are all under “cross-lingual/domain” columns in Table 2). - It would be interesting to report the learned optimal .alpha value for each different setting (at varying r or training size) to see if the authors’ intuition is met. On a related note, from the manuscript alone, it is not entirely clear if .alpha is a learnable parameter or a tunable hyper parameter -- by context I believe it is a model parameter. If they are tuned, authors need to report these values. <Nit> - Section 3.2.3, “... (GRAD) to enable adaptive weights for [each sample]” → I think it reads better with [each resource] or [each source and target], unless you meant .alpha_i for each sentence. - Section 3.2.5, “... recently, adversarial samples are [wisely] incorporated” → [widely] - Fonts for figures could be bigger.