# Cross-Lingual Named Entity Recognition Based on Attention and Adversarial Training
## 3 Model
### Adversarial Training Layer  
To better improve the cross-language word vector alignment, this paper constructs a shared semantic space from the source language to the target language. In a recent study, [PERSON] [21] explores unsupervised methods to learn cross-language word vectors and achieves comparable performance with supervised methods. In this paper, word-level adversarial training is performed to automatically align the word representations of the source and target languages.  
To construct the adversarial training, this paper first gives the pre-trained monolingual word vector target language \(V_{t}=\left\{v_{1}^{t},v_{2}^{t},\dots,v_{N}^{t}\right\}\ \in\ R^{N\times d_{t}}\)_and source language \(V_{s}=\left\{v_{1}^{s},v_{2}^{s},\dots,v_{M}^{s}\right\}\ \in\ R^{M\times d_{s}}\)_, where \(v_{i}^{t}\) and \(v_{i}^{s}\) are vector representations of words \(w_{i}^{t}\) and \(w_{i}^{s}\), respectively; t, s, N, and M indicate dictionary size; and \(d_{t}\), \(d_{s}\) indicate word vector dimensions. Then, a mapping method f is applied to map s to the same semantic space as t:  
\[\widetilde{V}_{s}=f(V_{s})=V_{s}U \tag{2}\]  
where \(U\) is a transformation matrix, \(\widetilde{V}_{s}\) is the word vector mapped by \(s\), and this paper uses singular value decomposition (SVD) to constrain the orthogonality of the transformation matrix \(U\) when generating the transformation matrix \(U\) in order to reduce the parameter search space:  
\[U=AB^{T},\ A\sum B^{T}=SVD(\widetilde{V}_{s}V_{s}^{T}) \tag{3}\]  
In order to optimize the mapping method without using additional bilingual information, a multi-layer perceptron \(D\) is introduced as a word discriminator in this paper. The role of the word discriminator is to generate a separate tensor by taking the target word vector and the mapping word vector as input features. \(D\big{(}w_{i}^{*}\big{)}\) is used to denote the probability that \(w_{i}^{*}\) comes from \(t\). This word discriminator is optimized by minimizing the binary cross-entropy loss:  
\[L_{dis}^{w}=-\frac{1}{I_{ts}}.\sum_{i=0}^{I_{ts}}(y_{i}\cdot log(D(w_{i}^{*}) )+(1-y_{i})\cdot log(1-D(w_{i}^{*}))) \tag{4}\]  
\[y_{i}=\delta_{i}(1-2\epsilon)+\epsilon \tag{5}\]  
When \(w_{i}^{*}\) is from the target language word vector, \(\delta_{i}=1\); otherwise, \(\delta_{i}=0\). \(I_{ts}\) denotes the number of words sampled together from the vocabulary of t and s, and \(\epsilon\) is the smoothed value for adding positive and negative labels. \(\Theta_{dis}=\left\{\theta_{D}\right\}\) is the parameter set.  
The total loss function consists of two components, a word discriminator \(D\) and a mapping method \(f\), which optimizes \(f\) by flipping word labels and optimally minimizing the total loss function:  
\[L_{f}^{w}\!=-\frac{1}{I_{L^{w}}}\cdot\sum\
olimits_{i=0}^{I_{L^{w}}}((1-y_{i}) \cdot log(D(w_{i}^{*}))+(y_{i})\cdot log(1-D(w_{i}^{*}))) \tag{6}\]  
\[y_{i}=\delta_{i}(1-2\epsilon)+\epsilon \tag{7}\]  
In adversarial learning, the network continuously minimizes the loss of the task discriminator in order to adversarially encourage the shared feature extractor to learn cross-linguistic information for both languages. After training, the task discriminator is unable to distinguish the word vectors of the two languages from the final shared features.