Introduction
Named Entity Recognition is a major natural language processing task that recognizes the proper labels such as LOC (Location), PER (Person), ORG (Organization), etc. Like words or phrase, being a sort of language constituent, named entities also benefit from better representation for better processing. Continuous word representations, known as word embeddings, well capture semantic and syntactic regularities of words BIBREF0 and perform well in monolingual NE recognition BIBREF1 , BIBREF2 . Word embeddings also exhibit isomorphism structure across languages BIBREF3 . On account of these characteristics above, we attempt to utilize word embeddings to improve NE recognition for resource-poor languages with the help of richer ones. The state-of-the-art cross-lingual NE recognition methods are mainly based on annotation projection methods according to parallel corpora, translations BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 and Wikipedia methods BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 .
Most annotated corpus based NE recognition tasks can benefit a great deal from a known NE dictionary, as NEs are those words which carry common sense knowledge quite differ from the rest ones in any language vocabulary. This work will focus on the NE recognition from plain text instead of corpus based NE recognition. For a purpose of learning from limited annotated linguistic resources, our preliminary discovery shows that it is possible to build a geometric space projection between embedding spaces to help cross-lingual NE recognition. Our study contains two main steps: First, we explore the NE distribution in monolingual case. Next, we learn a hypersphere mapping between embedding spaces of languages with minimal supervision.
Despite the simplicity of our model, we make the following contributions. First, for word embeddings generated by different dimensions and objective functions, all common NE types (PER, LOC, ORG) tend to be densely distributed in a hypersphere, which gives a better solution to characterize the general NE distribution rather than existing closed dictionary definition for NE. Second, with the help of the hypersphere mapping, it is possible to capture the NE distribution of resource-poor languages with only a small amount of annotated data. Third, our method is highly friendly to unregistered NEs, as the distance to each hypersphere center is the only factor needed to determine their NE categories. Finally, by adding hypersphere features we can significantly improve the performance of off-the-shelf named entity recognition (NER) systems.
Word Embeddings
Seok BIBREF2 proposed that similar words are more likely to occupy close spatial positions, since their word embeddings carries syntactical and semantical informative clues. For an intuitive understanding, they listed the nearest neighbors of words included in the PER and ORG tags under cosine similarity metric. To empirically verify this observation and explore the performance of this property in Euclidean space , we list Top-5 nearest neighbors under Euclidean distance metric in Table 1 and illustrate a standard t-SNE BIBREF12 2- $D$ projection of the embeddings of three entity types with a sample of 500 words for each type.
Nearest neighbors are calculated by comparing the Euclidean distance between the embedding of each word (such as Fohnsdorf, Belgian, and Ltd.) and the embeddings of all other words in the vocabulary. We pre-train word embeddings using the continuous skip-gram model BIBREF13 with the tool, and obtain multi-word and single-word phrases with a maximum length of 8, and a minimum word frequency cutoff of 3. The examples in Table 1 and visualization in Figure 1 demonstrate that the above observation suits well under Euclidean distance metric for NE recognition either for monolingual or multilingual situations.
Model
Encouraged by the verification of nearest neighbors of NEs still being NEs, we attempt to build a model which can represent this property with least parameters. Namely, given an NE dictionary on a monolingual, we build a model to describe the distribution of the word embeddings of these entities, then we can easily use these parameters as a decoder for any word to directly determine whether it belongs to a certain type of entity. In this section, we first introduce the open modeling from embedding distribution in monolingual cases, and then put forward the mapping of the distribution model between languages, and then use the mapping to build named entity dataset for resource-poor languages. Finally, we use the proposed named entity model to improve the performance of state-of-the-art NE recognition systems.
Open Monolingual NE Modeling
As illustrated is Figure 1, the embedding distribution of NEs is aggregated, and there exists a certain boundary between different types of NEs. We construct an open representation for each type of NEs – hypersphere, the NE type of any entity can be easily judged by checking whether it is inside a hypersphere, which makes a difference from the defining way of any limited and insufficient NE dictionary. The hypersphere can be expressed as follows:
$$E( X, O) \le r$$   (Eq. 9)
where E represents the adopted Euclidean distance, X is referred to any point in the hypersphere, $ O $ and $ r $ are the center vector and radius. For each entity type, we attempt to construct a hypersphere which encompass as many congeneric NEs as possible, and as few as possible inhomogeneous NEs, we use $F_1$ score as a trade-off between these two concerns. We carefully tune the center and radius of the hypersphere to maximize its $F_1$ score: we first fix the center as the average of all NE embeddings from known NE dictionaries, and search the best radius in $[minDist, maxDist]$ , where $minDist/maxDist$ refers to the distance between the center and its nearest/farthest neighbors; Then, we kick NEs which are far from the center with the distance threshold $q$ (much larger than the radius) to generate a new center; Finally, we tune the threshold $q$ and repeat the above steps to find the most suitable center and radius.
The mathematical intuition for using a hypersphere can be interpreted in a manner similar to support vector machine (SVM) BIBREF14 , which uses the kernel to obtain the optimal margin in very high dimensional spaces through linear hyperplane separation in Descartes coordination. We transfer the idea to the separation of NE distributions. The only difference is about boundary shape, what we need is a closed surface instead of an open hyperplane, and hypersphere is such a smooth, closed boundary (with least parameters as well) in polar coordinates as counterpart of hyperplane in Descartes coordinates. Using the least principle to model the mathematical objective also follows the Occam razor principle.
Figure 1 also reveals that the distribution of PER NEs is compact, while ORG NE distribution is relatively sparse. Syntactically, PER NEs are more stable in terms of position and length in sentences compared to ORG NEs, so that they have a more accurate embedding representation with strong strong syntax and semantics, making the corresponding word embeddings closer to central region of the hypersphere.
 Embedding Distribution Mapping
As the isomorphism characteristic exists between languages BIBREF3 , BIBREF15 , we can apply the distributional modeling for every languages in the same way. For a target language without an NE dictionary, its NE distribution can be obtained from a source language with known NE distributions by learning the transforming function between these two languages. We construct the transformation matrix $W$ via a set of parallel word pairs (the set will be referred to seed pairs hereafter) and their word embeddings $\lbrace X^{(i)}, Z^{(i)}\rbrace _{i=1}^m$ BIBREF3 , $\lbrace X^{(i)}\rbrace _{i=1}^m$ , $\lbrace Z^{(i)}\rbrace _{i=1}^m$ are the source and target word embeddings respectively. $W$ can be learned by solving the matrix equation $XW = Z$ . Then, given the source center vector ${ O_1}$ , the mapping center vector ${O_2}$ can be expressed as:
$${ O_2} = W^T{O_1}$$   (Eq. 11)
Actually, the isomorphism (mapping) between embedding spaces is the type of affine isomorphism by furthermore considering embedding in continuous space. The invariant characteristics of relative position BIBREF16 , BIBREF17 , BIBREF18 , BIBREF19 in affine transformation is applied to correct transformation matrix errors caused by limited amount of parallel word pairs (the set will be referred to seed pairs hereafter). As shown in Figure 2, the ratio of the line segments keep constant when the distance is linearly enlarged or shortened. Recall that point $Q$ is an affine combination of two other noncoincident points $Q_1$ and $Q_2$ on the line: $Q = (1-t)Q_1 + tQ_2 $ .
We apply the affine mapping $f$ and get: $f(Q) = f((1-t)Q_1 + tQ_2) = (1-t)f(Q_1) + tf(Q_2)$ Obviously, the constant ratio $t$ is not affected by the affine transformation $f$ . That is, $Q$ has the same relative distances between it and $Q_1$ and $Q_2$ during the process of transformation. Based on the above characteristic, for any point $X^{(i)}$ in the source space and its mapping point $Z^{(i)}$ , $X^{(i)}$ and $f(Q) = f((1-t)Q_1 + tQ_2) = (1-t)f(Q_1) + tf(Q_2)$0 cut off radiuses with the same ratio, namely, the ratio of the distance of these two points to their centers and their radiuses remains unchanged.
$$\frac{E( O_1, X^{(i)})}{r_1} = \frac{E( O_2, Z^{(i)})}{r_2}$$   (Eq. 15)
where $E$ represents the adopted Euclidean distance, ${O_1, O_2, r_1, r_2}$ are the centers and radii of hyperspheres. We convert the equation and learn the optimized mapping center ${O_2}$ and ratio $K$ via the seed pairs:
$${K = \frac{r_2}{r_1} = \frac{E( O_2, Z^{(i)})}{E( O_1, X^{(i)})}}$$   (Eq. 16)
$$\begin{aligned} E( O_2, Z^{(i)}) &= K * E( O_1, X^{(i)}) \quad r_2 &= K * r_1 \\ \end{aligned}$$   (Eq. 17)
Given the seed pairs $\lbrace X^{(i)}, Z^{(i)}\rbrace _{i=1}^m$ , the initialized center $O_2$ in Equation (3), the center $ O_1 $ and radius $ r_1 $ of the hypersphere in source language space, we may work out the optimized ratio $K$ , the mapping center $ O_2 $ and radius $ r_2 $ in target language space by solving the linear equation group (5).
Hypersphere features for NE Recognition 
The Euclidean distance between word and hypersphere centers can be pre-computed as its NE likelihood, which may provide informative clues for NE recognition. We only consider three entity types in our experiment, and the Euclidean distance which is represented as a 3- $D$ vector and referred to HS vector hereafter) is added to four representative off-the-shelf NER systems to verify its effectiveness. We feed HS vector into different layers of the neural network: (1) input layer $[x_k; c_k; HS]$ ; (2) output layer of LSTM $[h_k; HS]$ , where $x_k$ , $w_k$ and $h_k$ represent word embeddings, char embeddings and the output of LSTM, respectively. All of these models are based on classical BiLSTM-CRF architecture BIBREF20 , except that BIBREF21 replaces CRF layer with softmax. These four baseline systems are introduced as follows.
BIBREF22 concatenates ELMo with word embeddings as the input of LSTM to enhance word representations as it carries both syntactic and semantic information.
BIBREF21 uses distant supervision for NER task and propose a new Tie or Break tagging scheme, where entity spans and entity types are encoded into two folds. They first build a binary classifier to distinguish Break from Tie, and then learn the entity types according to their occurrence and frequency in NE dictionary. The authors conduct their experiments on biomedical datasets rather than standard benchmark, so we extract the NEs in training data as the domain-specific dictionary. This work creates a promising prospect for using dictionary to replace the role of training data.
BIBREF23 takes advantage of the power of the 120 entity types from annotated data in Wikipedia. Cosine similarity between the word embedding and the embedding of each entity type is concatenated as the 120- $D$ feature vector (which is called LS vector in their paper) and then fed into the input layer of LSTM. Lexical feature has been shown a key factor to NE recognition.
BIBREF24 passes sentences as sequences of characters into a character-level language model to produce a novel type of word embedding, contextual string embeddings, where one word may have different embeddings as the embeddings are computed both on the characters of a word and its surrounding context. Such embeddings are then fed into the input layer of LSTM.
Experiment
In this section, we evaluate the hypersphere model based on the three models introduced above: open monolingual NE modeling, embedding distribution mapping and refinement NE recognition.
Setup
In this experiment, we adopt pre-trained word embeddings from Wikipedia corpus. Our preliminary experiments will be conducted on English and Chinese. For the former, we use NLTK toolkit and LANGID toolkit to perform the pre-processing. For the latter, we first use OpenCC to simplify characters, and then use THULAC to perform word segmentation.
In order to make the experimental results more accurate and credible, we manually annotate two large enough Chinese and English NE dictionaries for training and test. Table 2 lists the statistics of Wikipedia corpus and the annotated data. Our dictionary contains many multi-word NEs in LOC and ORG types as accounted in the second column for each language in Table 2, while we only include single-word PER NEs in our dictionary, since the English first name and last name are separated, and Chinese word segmentation cuts most of the PER entities together. We pre-train quality multi-word and single-word embeddings and aim to maximize the coverage of the NEs in the dictionary. The pre-trained word embeddings cover 82.3% / 82.51% of LOC NEs and 70.2% / 63.61% of ORG NEs in English and Chinese, respectively. For other multi-word NEs, we simply calculate the average vector of each word embedding as their representations.
Monolingual Embedding Distribution
The NE distribution is closely correlated to the dimension of the embedding space, we train the word embeddings from 2- $D$ to 300- $D$ and search for the most suitable dimension for each NE type. For each dimension, we carefully tune the center and radius of the hypersphere using the method introduced in section 3.1 for maximize $F_1$ score, and select the dimension with maximize $F_1$ score. The most suitable dimension for ORG, PER, LOC are 16- ${D}$ /16- ${D}$ /24- ${D}$ (these dimensions will be used as parameters in the following experiments), respectively . We discover that in low-dimensional space, the distributions of NEs are better. In high dimensions, the curse of dimension could be the main reason to limit the performance.
Table 3 lists the final maximum $F_1$ score of three NE types. The results of the three types of NE are almost 50%, and PER type performs best. The main factor may be that PER NEs are represented as single-word in our dictionary, and word embeddings can better represents their meanings. The result also states that better representations for multi-word NEs which are not covered by the dictionary instead of the average of each word may help bring better results. Besides, the incompleteness of NE dictionaries and noises during pre-processing may cause a decrease on the performance. Overall, hypersphere model has shown been effectively used as the open modeling for NEs.
 Hypersphere Mapping
The following preparations were made for the mapping: $(i)$ A large enough NE dictionary in source (resource-rich) corpus; $(ii)$ A small amount of annotated seed pairs. We use $s$ to represent the number of seed pairs and $d$ to represent the number of unknown variables. With seed pair size $s < d$ , the matrix can be solved with much loose constraints, and $F_1$ score remarkably increases with more seed pairs. Once $s > d$ , the linear equation group will be always determined by strong enough constraints, which leads to a stable solution. Based on the characteristics, we only take two dozen of seed pairs on each type in following experiments. We combine human translation and online translation together for double verification for this small set of seed pairs. In this part, we utilize English and Chinese as the corpus of known NEs in turn, and predict the NE distribution of the other language.
Evaluation In order to quantitatively represent the mapping effect, we present a new evaluation method to judge the hypersphere mapping between English and Chinese:
$$\begin{aligned} P = \frac{V_i}{V_m} \quad R = \frac{V_i}{V_t} \quad F_1 = \frac{2 * P * R}{P + R} \end{aligned}$$   (Eq. 29)
where ${V_t, V_m, V_i}$ represent the volumes of the target, mapping and intersection hyperspheres. Due to the difficulty of calculating the volume of hyperspheres in high dimensions, we adopt Monte Carlo methods to simulate the volume BIBREF25 . we generate a great quantity of points in the embedding spaces, and take the amount of the points falling in each hypersphere as its volume.
Mapping between English and Chinese Table 4 shows the comparisons of cross-lingual named entity extraction performance. We use the unsupervised method proposed in BIBREF26 to generate cross-lingual embeddings. $k$ -NN and SVM are the same as monolingual cases in Table 3 except for the training set. $k$ -NN $_{150}$ and SVM $_{150}$ use 20% of the NEs in source language and 150 NEs (50 LOC, PER and ORG) in target language for training, while $k$ -NN $_{2500}$ and SVM $_{2500}$ use 20% of the NEs in source language and 2500 NEs (1000 LOC and PER, 500 ORG) in target language. $k$ -NN and SVM depend much on the annotated training set, requiring more than $1K$ training samples to provide a performance as our model offers. Due to the instability of ORG type in length, taking the average of each word embedding may disobey the syntactic and semantic regularities of ORG NEs, thereby undermines the multilingual isomorphism characteristics, which causes the inferior performance of our model on this type of NEs. This suggests that build better representations NEs for multi-word NEs may contribute to a better performance in our model.
Mapping to truly Low-resource Language We build named entity dataset for a truly resource-poor language, Indonesian, and manually examine the nearest words to the hypersphere center for 'gold-standard' evaluation. We take English as the source language, the settings of the dimension $D$ and the number of seed pairs $s$ are the same as the above experiments between Chinese and English. From the results listed in Table 5, we can see that even the precision of the top-100 NEs are 0.350 $F_1$ /0.440 $F_1$ /0.310 $F_1$ , respectively, which proves the this distribution can indeed serves as a candidate NE dictionary for Indonesian.
[9] The authors of BIBREF24 publish an updated results (92.98) on CoNLL-2003 dataset in https://github.com/zalandoresearch/flair/issues/206 on their 0.3.2 version, and this is the best result at our most try. [10] This is the reported state-of-the-art result in their github. [11]We use the same parameters as the authors release in https://github.com/zalandoresearch/flair/issues/173 and obtain the result of 89.45 on ONTONOTES 5.0 dataset.
Off-the-shelf NE Recognition Systems
To evaluate the influence of our hypersphere feature for off-the-shelf NER systems, we perform the NE recognition on two standard NER benchmark datasets, CoNLL2003 and ONTONOTES 5.0. Our results in Table 6 and Table 7 demonstrate the power of hypersphere features, which contribute to nearly all of the three types of entities as shown in Table 6, except for a slight drop in the PER type of BIBREF22 on a strong baseline. HS features stably enhance all strong state-of-the-art baselines, BIBREF22 , BIBREF21 and BIBREF23 by 0.33/0.72/0.23 $F_1$ point and 0.13/0.3/0.1 $F_1$ point on both benchmark datasets, CoNLL-2003 and ONTONOTES 5.0. We show that our HS feature is also comparable with previous much more complicated LS feature, and our model surpasses their baseline (without LS feature) by 0.58/0.78 $F_1$ point with only HS features. We establish a new state-of-the-art $F_1$ score of 89.75 on ONTONOTES 5.0, while matching state-of-the-art performance with a $F_1$ score of 92.95 on CoNLL-2003 dataset.
Related Work
In recent years, word embeddings have also been used as a feature to enhance the NE recognition, with the revealing of linguistic features in morphological, syntactic and semantic perspective. BIBREF1 clustered the word embeddings and combined multiple cluster granularities to improve the NE recognition performance. Our work likewise use word embeddings to help NE recognition, we make use of the characteristic that syntactically and semantically s are more likely to be neighbors in embedding spaces and construct a hypersphere model to encompass NEs.
Cross-lingual knowledge transfer is a highly promising work for resource-poor languages, annotation projection and representation projection are widely used in NE recognition BIBREF27 , BIBREF5 , BIBREF4 , BIBREF28 , BIBREF29 , BIBREF30 . These works put forward inconvenient requirements for parallel or comparable corpora, a large amount of annotated or translation data or bilingual lexicon. Different from any existing work to the best of our knowledge, this is the first work that merely uses isomorphic mappings in low-dimensional embedding spaces to recognize NEs, and we introduce a mathematically simple model to describe NE embedding distribution from visualization results, showing it works for both monolingual and cross-lingual situations.
Conclusion
Named entities being an open set which keeps expanding are difficult to represent through a closed NE dictionary. This work mitigates significant defects in previous closed NE definitions and proposes a new open definition for NEs by modeling their embedding distributions with least parameters. We visualize NE distributions in monolingual case and perform an effective isomorphism spaces mapping in cross-lingual case. According to our work, we demonstrate that common named entity types (PER, LOC, ORG) tend to be densely distributed in a hypersphere and it is possible to build a mapping between the NE distributions in embedding spaces to help cross-lingual NE recognition. Experimental results show that the distribution of named entities via mapping can be used as a good enough replacement for the original distribution. Then the discovery is used to build an NE dictionary for Indonesian being a truly low-resource language, which also gives satisfactory precision. Finally, our simple hypersphere features being the representation of NE likelihood can be used for enhancing off-the-shelf NER systems by concatenating with word embeddings and the output of BiLSTM in the input layer and encode layer, respectively, and we achieve a new state-of-the-art $F_1$ score of 89.75 on ONTONOTES 5.0 benchmark. In this work, we also give a better solution for unregistered NEs. For any newly emerged NE together with its embedding, in case we obtain the hypersphere of each named entity, the corresponding named entity category can be determined by calculating the distance between its word embedding and the center of each hypersphere.

Question: What is their model?
Answer: cross-lingual NE recognition

Question: Do they evaluate on NER data sets?
Answer: Yes