Venue: NAACL 2021

# Field Embedding: A Unified Grain-Based Framework for

Word Representation

Junjie Luo\({}^{1,2}\) , Xi Chen\({}^{1}\)+, Jichao Sun\({}^{1}\)+, Yueja Xiang\({}^{1}\), Ningyu Zhang\({}^{3}\), Xiang Wan\({}^{4}\)

\({}^{1}\)Tencent Jarvis Lab \(~{}^{2}\)University of Maryland

\({}^{3}\)Zhejiang University \(~{}^{4}\)Shenzhen Research Institute of Big Data

jjluo@umd.edu zhangningyu@zju.edu.cn wanxiang@sribdb.cn {jasonxchen,jichaosun,yuejiaxiang}@tencent.com

The first three authors contributed equally.Xi Chen is the corresponding author.

###### Abstract

Word representations empowered with additional linguistic information have been widely studied and proved to outperform traditional embeddings. Current methods mainly focus on learning embeddings for words while embeddings of linguistic information (referred to as grain embeddings) are discarded after the learning. This work proposes a framework _field embedding_ to jointly learn both word and grain embeddings by incorporating morphological, phonetic, and syntactical linguistic fields. The framework leverages an innovative fine-grained pipeline which integrates multiple linguistic fields and produces high-quality grain sequences for learning supreme word representations. A novel algorithm is also designed to learn embeddings for words and grains by capturing information that is contained within each field and that is shared across them. Experimental results of lexical tasks and downstream natural language processing tasks illustrate that our framework can learn better word embeddings and grain embeddings. Qualitative evaluations show grain embeddings effectively capture the semantic information.

## 1 Introduction

Distributed word representation, also named as word embedding, represents each word as a vector in a continuous vector space. Due to its strong ability of encoding semantic information, word embedding is useful in many downstream NLP tasks, such as text classification Wierting et al. (2016); Yin et al. (2016), named entity recognition (NER) Collobert et al. (2011); Sun et al. (2015), etc. Classic approaches mainly treated words as atomic tokens, such as WordVec Mikolov et al. (2013b, a) and GloVe Pennington et al. (2014). Recently, many researchers introduced subword information to learn advanced word embeddings for different languages, including English Bojanowski et al. (2017) and Chinese Yu et al. (2017); Cao et al. (2018). In this paper, we refer to subword types as linguistic fields and symbols representing subwords as grains. For example, the field _letter_ grain sequence of word _wisdom_ is [w, i, s, d, o, m]. Elements in the sequence are _letter_ grains which are from the _letter_ vocabulary, i.e., the alphabet table. Fig. 1 shows more examples of linguistic fields and grains.

However, though huge progress has been achieved, there are many challenges or limitations for fully exploiting linguistic fields' potential on learning advanced embeddings. The first challenge is producing semantically meaningful representations for input words. Such representations rely on (a) broad linguistic fields and (b) high-quality grain sequences. For fields, only morphological fields were studied, such as _letter_ in English Bojanowski et al. (2017), and _component_ in Chinese Yu et al. (2017). However, linguistics studies revealed that phonetic and syntactical fields contain rich semantic information Beaver et al. (2007), whose utility was not fully studied before. For grain sequences, current methods only produced coarse grain sequences, whose grains seldom carry information associated with the original word. For example, grains from [w, i, s, d, o, m], the _letter_ grain sequence of word _wisdom_, are simple and less meaningful letters.

Second, a customized algorithm is required to

Figure 1: Field and grain sequence examples for */ (wisdom) and _wisdom_. ** notations** indicates the hyperfield.



model both the uniqueness of each field and the relationship among fields, given the increasingly available linguistic fields as the new information. Besides their uniqueness, fields have strong correlations with each other. We can find many morpheme-syntax pairs, such as -tion:Noun and -ious:Adj and morpheme-phoneme pairs: sounds of \(\nexistsq\) (mother) and \(\nistq\) (code), which are derived from the pair \(\xrightarrow{\mathit{T}_{J}}\) (horse):ma. However, past methods might fail to capture holistic information carried across fields Chen et al. (2015); Cao et al. (2018). They simply put all fields together and immigrated classical word2vec algorithms, which ignored such inter-field information.

Furthermore, the value of grain embeddings on NLP tasks has not been comprehensively evaluated. Similar to word embeddings, we introduce grain embeddings which represent each grain with a semantic vector. Past work focused on word embeddings but paid little attention to learning and evaluating grain embeddings for linguistic fields. Whether grain embeddings can convey semantics and benefit NLP tasks is still not systematically studied.

To solve the above challenges, we propose a _field embedding_ framework to jointly learn word and grain embeddings simultaneously. It can flexibly integrate any combination of linguistic fields. Our contributions are follows:

(1) A fine-grained pipeline (a) takes any combination of various linguistic fields, including morpheme, phoneme, and syntax, as the input, and (b) includes n-gram and grain dropping to generate high-quality grain sequences as semantically meaningful and complete representations for input words.

(2) A novel algorithm is proposed with the motivation of ubiquitous linguistic phenomena. Its loss function generates two kinds of gradients to model information contained within each field and that shared across multiple fields separately. This brings holistic information to improve the embedding quality.

(3) Extensive experimental results illustrate that our framework yields supreme word and grain embeddings in various NLP tasks. Our framework learns better word embeddings than previous methods in both lexical tasks and downstream tasks. Moreover, our learned grain embeddings outperform word embeddings in downstream tasks, such as text classification and NER. Furthermore, qualitative evaluations show that grain embeddings can effectively capture semantic information.

It is the first toolkit that can measure the effectiveness of various fields and their combinations in learning word and grain embeddings. Its simplicity and compatibility spare the laborious and time-consuming developments and evaluations of new multi-field models. The code and data will be released on Github.

## 2 Background and Related Works

**Field and Grain** A sentence is a list of words. A field describes one linguistic aspect of words. For example, in Chinese, _component_ and _stroke_ are morphological fields which describe the word shape, _pinyin_, a phonetic field, describes pronunciations. These are subfields, which are determined exclusively by the word. In contrast, hyperfields refer to the linguistic fields determined by both the word and its context, such as _part-of-speech (POS)_, a syntactical field. A simple and efficient way to represent field information is using sequences of symbols. Symbols are referred to as grains and sequences as grain sequences, such as _wisdom_'s _word root_ grain sequence is [wise-, -dom].

**Static Embeddings** Static embeddings present each word (or grain) with a semantic vector independent of its contexts. This work focuses on exploiting linguistic potential for learning static word and grain embeddings. Past works achieved huge progress in introducing linguistic fields for advanced static word embeddings. For English, one typical linguistic field is _letter_, which has been exploited to improve word embeddings Bojanowski et al. (2017). For Chinese, subword fields, including _character_, _component_, and _stroke_, convey fruitful semantic information Wieting et al. (2015); Liu et al. (2017) and have been studied by CWE Chen et al. (2015), JWE Yu et al. (2017), and cw2vec Cao et al. (2018) separately. The above methods adopted shallow but efficient structures. In contrast, many methods introduced deep neural networks in learning static embeddings Kim et al. (2018); Cao and Lu (2017). However, they cost huge computational resources and yielded limited improvements. To keep the framework straightforward and efficient, this work adopts a shallow structure.

**Dynamic Embeddings** Dynamic embeddings are trained as deep language models and represent 

a whole sentence with contextual information. The dynamic embeddings, such as ELMo Peters et al. (2018), and BERT Devlin et al. (2018), achieved state-of-the-art performances in many NLP tasks. This work compares our grain embeddings with them on downstream tasks. However, we did not evaluate them with lexical tasks, as they cannot produce word vector without contexts directly.

## 3 Model Description

We use \(C\) to denote the training corpus, \(V\) the word vocabulary, and \(F\) the set of selected linguistic fields. For a field \(f\in F\), we generate its n-gram grain vocabulary \(V_{f}\) by scanning the words in the whole corpus. \(G_{f}(w)=[g_{1},g_{2},\dots,g_{n}]\) is the grain sequence of word \(w\) in field \(f\). Data points fed into embedding models are pairs of a target word \(w_{t}\) and its context word set \(S(w_{t})\), or \(S\) for simplicity. Take the sentence _The fox runs after cats_ as an example and suppose _runs_ is the target word. By applying _cbow_, the data points \((S,w_{t})\) will be ([The, fox, after, cats], runs). By applying _skip-gram_, there are four \((S,w_{t})\)s: ([_ctx_], runs) for _ctx_ in [The, fox, after, cats].

The _cbow_ structure is shown in Fig. 2. It contains three layers: the input, projection, and prediction layer. For a data point \((S,w_{t})\), multi-field inputs of \(S\) in the input layer are fed to the projection layer and become projection vectors: \(P_{0}\) to \(P_{|F|}\). Each projection enters the prediction layer to predict \(w_{t}\) and get a prediction loss. The summation of these losses is the total loss. Model's parameters include several grain embedding matrices in the projection layer and a word embedding matrix in the prediction layer. We represent word embeddings as \(E_{W}\) of size \(|V|\times d\), where \(d\) is the embedding size. For each field \(f\), its grain embeddings \(E^{f}_{G}\) is of size \(|V_{f}|\times d\). Word embeddings \(E_{W}\) vectorize the target word \(w_{t}\) while grain embeddings \(E^{f}_{G}\) only vectorize grains from context words in \(S(w_{t})\).

### Fine-Grained Pipeline

In the projection layer, we design a fine-grained pipeline that consists of n-gram and grain dropping to produce high-quality grain sequences. Fig. 3 shows its mechanism.

**N-Gram** Compared to the word vocabulary \(V\), sizes of \(V_{f}\) and \(E^{f}_{G}\) are small, which may lead to the underfitting problem. For each field \(f\), we generate n-gram grains to enlarge its grain vocabulary \(V_{f}\). By increasing the grain vocabulary size, it enlarges each word's grain sequence for a higher capacity of carrying linguistic knowledge. As an instance, the word \(\frac{\text{\tiny{$\mathfrak{M}$}}}{\text{\tiny{$\mathfrak{M}$}}}\) (wisdom) contains _component_\(\mathcal{K}\) (arrow), \(\square\) (mouth), and \(\blacksquare\) (day). These components are not relevant to the semantics of \(\frac{\text{\tiny{$\mathfrak{M}$}}}{\text{\tiny{$\mathfrak{M}$}}}\) (wisdom). Without n-gram grains, \(G_{f}(w)\) is a short sequence which hardly catches enough information, whereas n-gram grains introduce more relevant grains which carry rich semantic information. For example, after including 2-gram grains, we have \(\mathcal{K}\ 

process.

As shown in Fig. 3, after the pipeline process, the word "\(\Uparrow\)'s coarse _component_ grain sequence is updated to , which carries semantically meaningful new grains for the word . The case of English word _wisdom_ is also the same. In short, n-gram and grain dropping help generate a better grain sequence \(G_{f}(w)\), which will be proved to be crucial to enhance the quality of embeddings. Afterwards, we can get projection vectors. For field \(f\), we can represent a word \(w\) by averaging all of its grain vectors to get its word projection vector: \(p_{f}(w)=\frac{1}{|G_{f}(w)|}\sum_{g\in G_{f}(w)}E^{f}_{G}(g)\), where \(E^{f}_{G}(g)\) indicates the vector of \(g\) from \(E^{f}_{G}\). Then, we calculate the context projection vector \(P_{f}(S)\):

\[P_{f}(S)=\frac{1}{|S|}\sum_{w\in S}\frac{1}{|G_{f}(w)|}\sum_{g\in G_{f}(w)}E^{ f}_{G}(g).\]

### Loss Function

In the prediction layer, our proposed customized algorithm contains a novel loss function which is motivated by the following linguistic phenomena. First, one field represents a linguistic attribute, such as morheme describing shape while phoneme indicating sound. Therefore, each field contains its corresponding unique linguistic information. Moreover, fields have strong connections with each other. We can easily find many morpheme-syntax pairs, such as -tion:Noun and -ious:Adj. The morpheme-phoneme pairs are also ubiquitous: sounds of \(\Uparrow\) (sketch) and \(\Uparrow\) (sir) are from the pair \(\Uparrow\) (apply):shen. To better model the above phenomena, we design a novel loss function to learn linguistic information contained within each field and shared across multiple fields. Next, we will show the loss function design and how it learns the information.

After calculating field projections \(P_{f}(S)\) for all \(f\in F\), we obtain the content projection of \(S\) that defined as: \(P_{0}(S)=\frac{1}{|F|}\sum_{f\in F}P_{f}(S)\). The objective of this problem is to minimize the negative log-likelihoods of the conditional predictive probability for a target word \(w_{t}\) with its context word set \(S(w_{t})\):

\[L_{fe}(w_{t})=\phi(w_{t}|P_{0}(S))+\sum_{f\in F}\phi(w_{t}|P_{f}(S)) \tag{1}\]

The negative log-likelihood of conditional probability is defined by the negative logarithm of softmax function.

\[\phi(w_{t}|P)=-log\frac{exp(P^{T}E_{W}(w_{t}))}{\sum_{j=1}^{|V|}exp(P^{T}E_{ W}(w_{j}))} \tag{2}\]

where \(P\) represents the corresponding context projection. In practice, we adopt an optimization method based on the negative sampling and the standard gradient descend. Negative sampling is to replace the expensive denominator in Eq. (2) with a set of negative sampled words based on a frequency distribution.

\[\begin{split}&\phi(w_{t}|P)=-[\text{log}\,\sigma(P^{T}E_{W}(w_{ t}))+\\ &\lambda\mathbb{E}_{w_{ng}\sim\mathcal{D}}\,\text{log}\,\sigma(- P^{T}E_{W}(w_{ng})]\end{split} \tag{3}\]

where \(\sigma\) is a sigmoid function, \(\lambda\) is the number of negative samples, \(\mathbb{E}_{w_{ng}\sim\mathcal{D}}[\cdot]\) represents expectation, and the negative sampled word \(w_{ng}\) belongs to word frequency distribution \(\mathcal{D}\). Given a specific corpus \(C\), the objective likelihood is \(\mathcal{L}(C)=\sum_{w_{t}\in C}L_{fe}(w_{t})\).

Our loss function \(L_{fe}\) in Eq. (1) contains two terms. For the first term, the gradient of \(P_{0}\), \(grad_{0}\), is back propagated to every grain across whole fields. This gradient \(grad_{0}\) can be interpreted as updated linguistic information sharing by all the fields. For the second term, the gradient of field projection \(P_{f}\), \(grad_{f}\), only updates the grain vectors within the specific field \(f\). This gradient \(grad_{f}\) can be interpreted as the unique linguistic information of field \(f\). In this case, the gradient that updates each grain in field \(f\) is \(grad_{f}+\frac{1}{|F|}grad_{0}\), which contains both unique and shared linguistic information. Existing methods mainly used only (a) the first term, such as the usage of \(L_{w2v}=f(w_{t}|P_{0})\) in word2vec, or (b) the second term, \(L_{jwe}=\sum_{f\in F}f(w_{t}|P_{f})\) in JWE. These methods chose either shared or field-specific information, which might be not complete enough for learning embeddings.

## 4 Training and Evaluations

### Training Setups

**Training Corpus** We adopt the benchmark corpus, both Chinese and English Wikipedia data, to train embeddings with our framework. For Chinese, the segmentation tool is _jieba_, which was widely used in Chinese NLP works Li et al. (2019). We set the minimal word frequency as 10, obtaining 390,106 unique words. We set the n-grams of _character_ and _POS_ as 1 and others as 4. The first 10,000 grains 

ordered by frequency are kept. For English corpus, we set the minimal word frequency as 30, which yields 649,068 unique words. We extend _letter_ and _phoneme_ to 3-gram, and set _POS_ to 1-gram. First 20,000 common grains are kept.

**Baselines** To assess the effectiveness of our framework, we compare it with several state-of-the-art algorithms. **Word2vec** only uses the _word_ itself as a field, and is an effective and efficient toolkit for learning word embedding. It adopts the \(L_{w2v}\) to train the embeddings. **CWE** combines both _word_ and _character_ as fields and trains embeddings in a _cbow_ structure. It also adopts the \(L_{w2v}\). **JWE** incorporates _word_, _character_, and _component_ with the _cbow_ structure and adopts the \(L_{jwe}\). **ew2vec** uses n-gram _stroke_ to train embeddings in a _skip-gram_ method and adopts the \(L_{w2v}\).

**Hyperparameters** For a fair comparison, each word and grain embedding is of 200 dimensions for all algorithms. We set the window size and iteration to 5, the initial learning rate to 0.025, and the negative samples to 10.

### Evaluation Tasks

In the task evaluation part, lexical tasks are conducted to evaluate word embeddings and the effectiveness of new linguistic fields, fine-grained pipeline, and our novel loss function. Downstream tasks are conducted to evaluate performances of grain embeddings compared to word embeddings. Qualitative analysis is used to validate semantic information in grain embeddings. All task datasets are widely used in previous word embedding works.

**Lexical Evaluations** Lexical evaluations include word similarity and word analogy, which are widely applied to evaluate the quality of word embeddings.

(a) _Word Similarity_ This task evaluates the model's ability of capturing the semantic relevance between given word pairs. We adopt datasets Sim240 and Sim297 from Chen et al. (2015b) for word similarity tasks to evaluate Chinese word embeddings, and use Sim353 from Mikolov et al. (2013a) for English word embeddings. For each pair of words in each dataset, a human-labeled score is provided. We compute the cosine similarity of each word pair and use the Spearman correlation to measure the quality of word embeddings.

(b) _Word Analogy_ This task evaluates whether the word embeddings capture the linguistic relationship between word pairs. Given three words like Berlin, Germany, Paris, the model should infer that the most similar word vector vec(France) with vec(Germany)-vec(Berlin)+vec(Paris). We adopt the Chinese dataset provided by Chen et al. (2015b), and English datasets from Google Mikolov et al. (2013a) and MSR Mikolov et al. (2013c).

**Downstream Task** In downstream tasks, we use our embeddings to represent words in a text or sentence as input features. Both word embeddings and grain embeddings can construct a word representation for a word \(w\). By word embeddings \(E_{W}\), a word \(w\) can be represented as \(E_{W}(w)\). By grain embeddings, the representation of word \(w\) can be constructed by concatenating all word projection \(p_{f}(w)\) together, where \(f\in F\) are available fields. In downstream tasks, the learned embeddings are frozen and not updated in training phases.

(a) _Text Classification_ We follow cw2vec and select five topics in the Chinese dateset FudanCorpus and obtain 5,885 texts. For English, we use NewsGroup and obtain 20 topics as well as 18,756 texts. We average word representation of words in a text as its input feature vectors. We build the classifier by using SVM in _sklearn_ and utilize five-fold cross validation to obtain accuracy scores.

(b) _Named Entity Recognition_ For Chinese, we adopt Boson, which contains 19,214 sentences and five entity categories and randomly separate it into train, validation, and test parts by 8:1:1. For English, we use CoNLL2003, which contains 16,477 sentences, 4 entity categories, and its own dataset segmentation. We develop a CRF model Lafferty et al. (2001) based on PyTorch as the classifier. We adopt a simple Embed-CRF structure to evaluate the embedding quality and a complicated BiLSTM-CRF to validate both static and dynamic embeddings.

**Qualitative Evaluation** It is based on the vectors of selected _character_, _component_ and _pinyin_ grains from learned field grain embeddings. We evaluate their top similar words from the learned embeddings, which are retrieved based on the cosine similarity.

## 5 Experimental Result

The fields and their logograms we used in this work are: _word_ W, _POS_ Pos; if Chinese, _character_ H, _component_ C, _stroke_ S, _pinyin_ P; if English, _letter_ C, _phoneme_ P 



### Lexical Evaluation

The lexical tasks evaluate word embeddings in terms of different field combinations, using pipeline or not, and loss functions to verify the effectiveness of our proposed methods. Results in Table 1 show that our proposed framework achieves best performances for different tasks. For instance, in Chinese Sim240 task, our model W.Pos gets the best similarity score 58.08%, with a 4.64% increase from best baseline method JWE. Next, we analyze sources of the improvements.

**Linguistic Field** In Table 1, for both Chinese and English, our models which integrate phoneme and syntax fields perform better on three different tasks and outperform other existing models. For example, W.C.P.Pos achieves best performances in both Chinese Sim297 (1.93% increase from the best baseline JWE) and English Sim353 (3.04% increase from the best baseline CWE). This supports phoneme and syntax fields carry new linguistic fields which previously widely used morphological fields do not contain. Moreover, putting all fields together to train embeddings does not guarantee that the learned word embeddings can achieve best performances for all tasks, such as the Chinese W.C.P.Pos model in Table 1. This indicates some linguistic fields bring more noise than semantic information for the corresponding task. Instead, finding the best field combination to train embeddings for specific tasks is more important. Our proposed framework makes it easy to explore the best field combination for each specific task.

**Fine-Grained Pipeline** Fig.4 verifies the fine-grained pipeline's effectiveness. Given fields and loss function of the model, leveraging the whole fine-grained pipeline including n-gram and grain dropping can lead to advanced word embeddings. For example, in the W.H.C model, the fine-grained pipeline yields a 9.13% increase in the Chinese Analogy task. Beyond this, Fig. 5 further validates the effectiveness of grain dropping. In the W.H.P model, grain dropping achieves a 1.57% improvements in Analogy task. The improvements demonstrate that our fine-grained pipeline does produce high-quality grain sequences and can successfully capture more linguistic information that previous works missed.

**Loss Function** Fig. 6 asserts the effectiveness of our loss function. \(L_{fe}\) loss function outperforms other loss functions in all tasks. For example, in Sim240 task, \(L_{fe}\) achieves 4.49% increase compared to \(L_{w2v}\) and 3.04% increase compared to \(L_{jwe}\). It's the same for Sim297 and Analogy. Compared with \(L_{w2v}\) and \(L_{jwe}\), \(L_{fe}\) considers prominent linguistic phenomena such as morpheme-phoneme pairs. Therefore, it successfully captures the within-field and crossing-field information.

### Downstream Task

We conduct both Chinese and English downstream tasks, including text classification and NER, to test the performances of _word embeddings_ and _grain embeddings_.

**Text Classification** Table 5 shows with more linguistic fields, both word and grain embeddings gain performance improvements on text classification tasks. This reveals that phoneme and syntax fields can improve the quality of both word and grain embeddings. Moreover, grain embeddings \(E_{G}\) always outperform word embeddings \(E_{W}\). In NewsGroup task, \(E_{G}\) of W.C.P.Pos exceed \(E_{W}\) with 3.08%. Furthermore, compared with word embeddings, grain embeddings make larger improvements with more fields. For exam

\begin{table}
\begin{tabular}{c|c c c c} \hline \hline  & **Method** & **Sim240** & **Sim297** & **Analogy** \\ \cline{2-5}  & word2vec & 48.16 & 58.03 & 72.4 \\  & CWE & 50.8 & 55.33 & 33.37 \\ Chinese & JWE & 53.44 & 58.95 & 66.4 \\  & cw2vec & 49.88 & 48.42 & 39.35 \\ \cline{2-5}  & W.Pos & **58.08** & 59.24 & 78.8 \\  & W.C.Pos & 56.66 & 59.6 & **79.43** \\  & W.C.P.Pos & 56.53 & **60.88** & 79.12 \\ \hline \multirow{4}{*}{English} & **Method** & **Sim353** & **AnaMSR** & **AnaGoogle** \\ \cline{2-5}  & word2vec & 65.14 & 57.46 & 65.12 \\ \cline{1-1}  & CWE & 65.47 & 57.35 & 64.81 \\ \cline{1-1} \cline{2-5}  & W.C & 68.38 & **62.16** & **70.09** \\ \cline{1-1}  & W.P.Pos & 68.32 & 61.51 & 70.02 \\ \cline{1-1}  & W.C.P.Pos & **68.51** & 59.01 & 69.5 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Performance on lexical tasks achieved by word embeddings generated by the proposed method and state-of-the-art methods. Spearman correlation coefficient is presented in percentage (%).

\begin{table}
\begin{tabular}{l l|r r|r} \hline \hline \multicolumn{1}{c}{\multirow{2}{*}{**Method**}} & \multicolumn{2}{c}{**NewsGroup (English)**} & \multicolumn{2}{c}{**Fudan (Chinese)**} \\ \cline{2-5}  & \(E_{W}\) & \(E_{G}\) & \(E_{W}\) & \(E_{G}\) \\ \hline Word2Vec & 69.48 & – & 93.88 & - \\ W.C & 72.98 & 75.36 & 94.24 & 94.55 \\ W.C.P & 73.03 & 76.82 & 93.92 & 94.61 \\ W.C.P.Pos & 77.04 & **80.12** & 94.26 & **95.00** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Comparison of the proposed method with Word2Vec on the text classification tasks. Accuracy is presented in percentage (%).



ple, from W.C to W.C.P.Pos, \(E_{W}\) achieve a 4.06% increase while \(E_{G}\) achieves a 4.76% increase in NewsGroup task. Such additional improvement derives from additional linguistic information that is not included in word embeddings. This strongly indicates that grain embeddings carry more linguistic information than the associated word embeddings and can be a better alternative to word embeddings.

**Named Entity Recognition** As to NER performances of Table 3, a similar pattern is observed to that of text classification. It demonstrates that more linguistic fields benefit the NER tasks. For example, from W.C.P to W.C.P.Pos, grain embeddings gain 4.91% and 6.41% improvements in CoNLL2003 and Boson tasks. The reason for the improvements is that the hyperfield _POS_ carries the part of a sentence' s syntactical information, which is crucial in sequence labeling tasks. This illustrates the hyperfield's significance in learning embeddings. Moreover, it shows grain embeddings outperform word embeddings. For instance, in W.C.P.Pos model, F1 scores of grain embeddings exceed that of word embeddings with 8.48% and 6.35% in CoNLL2003 and Boson tasks.

To further prove that our embeddings are effective with complicated neural networks, we adopt Embed-BiLSTM-CRF and conduct experiments on CoNLL2003. Besides static embedding Word2Vec, dynamic embedding methods, such as ELMo and BERT, are also listed as baselines for comparison, as shown in Table 4. In terms of static embeddings, additional fields still benefit the task, with around 0.54% increase per field from Word2Vec to W.C.P.Pos. This shows that, even in complex neural networks, grain embeddings are superior to word embeddings, and phoneme and syntax are useful.

For dynamic embeddings, though it is marginally inferior to BERT, our W.C.P.Pos is better than EMLo by 0.12%. It indicates our framework exploits the potential of static embeddings with multiple fields which surpasses the relatively shallow dynamic embeddings. This also suggests the potential of introducing multi-fields to dynamic embeddings. Our static embeddings also enjoy other advantages. The model's structure is simple and straightforward and parameter size is small. It requires less corpus and resources to train compared to BERT, which is a complicated deep neural networks. In downstream tasks, though BERT outperforms our model, it bears expensive costs of model complexity and computational resources. Moreover, the dynamic embeddings cannot represent the independent word or gain, whereas our model yields high quality representations for them and achieves best performances in lexical tasks.

 



### Qualitative Evaluation

We evaluate the embeddings' abilities to uncover the semantic relatedness of words, characters, and components through case studies based on a model trained with _character_, _component_, and _pinyin_. Taking \(\mathbb{H}\) (apply) as an example, which is also an ancient state name and a popular last name in China, it can be a Chinese character or word. We list its closest words from word embeddings in Table 5 where we treat \(\mathbb{H}\) as a character and a word. When it is a character, most of the closest words are semantically related to _appply_. When it acts as a word, the closest words related to country name and last name meanings in \(\mathbb{H}\). For example, \(\mathbb{K}\) (Zhao), \(\mathbb{K}\) (Yin) are ancient state names and last names, and \(\mathbb{K}\) (Duke Ding) \(\mathbb{F}\) \(\mathbb{G}\) \(\mathbb{L}\) (Duke Zhao of Jin) are dukes in ancient China. This reveals that grain embeddings can supplement the word embeddings for a more complete semantic representation.

We further take the _component_\(\mathcal{Y}^{+-}\) (illness) as an example and Table 5 shows its closest characters and words. All of the closest characters and words are semantically related to the _component_\(\mathcal{Y}^{+-}\). Most of them are related to diseases, symptoms and other medical terms, such as words \(\mathbb{K}\) (disease), \(\mathbb{K}\) (infection) and characters \(\mathbb{H}\) (symptom), \(\mathbb{K}\) (score). Most of them contain _component_\(\mathcal{Y}^{+-}\), but \(\mathbb{H}\) (gangrene), \(\mathbb{K}\) (suffer), and \(\mathbb{K}\) (infection) without \(\mathcal{Y}^{+-}\) also share the similar semantics. Moreover, we study _pinyin_ t-ong, sound of \(\mathbb{H}\) (pain), and list its closest words and characters in Table 5 and observe a similar phenomenon to \(\mathcal{Y}^{+-}\). These closest characters have similar semantic meaning with \(\mathbb{H}\) (pain), whose pinyin is t-ong, such as words \(\mathbb{K}\) (pain), \(\mathbb{K}\) (headache), and characters \(\mathbb{K}\) (sore), \(\mathbb{K}\) (paralysis). The qualitative analysis shows that our proposed models leverage both external context co-occurrence information and internal morphological and phonetic information. The medical information stored in above grain embeddings could be utilized for clinical NER tasks.

## 6 Conclusion

We propose a flexible _field embedding_ framework to jointly learn both word and grain embeddings by incorporating morphological, phonetic, and syntactical linguistic fields simultaneously. Our proposed framework leverages an innovative fine-grained pipeline, including n-grams and grain dropping, as well as a novel loss function to capture the information contained within each linguistic field and shared across multiple fields. By introducing phonetic and syntactical linguistic fields and leveraging our fine-grained pipeline and loss function, our framework is capable of learning better word embeddings in terms of word similarity and analogy. Furthermore, we systemically investigate the effectiveness of grain embeddings and provides the evidence that grain embeddings can be a better alternative to word embeddings for word representations. Experimental results show that grain embeddings outperform word embeddings in several downstream NLP tasks, such as text classification and named entity recognition. The qualitative analysis illustrates that grain embeddings can effectively capture semantic information.

## Acknowledgments

We would like to thank Prof. Jordan Boyd-Graber from University of Maryland and Dr. Yefeng Zheng from Tencent for their helpful suggestions. This work was supported by the Key-Area Research and Development Program of Guangdong Province [2020B0101350001].

## References

* Beaver et al. (2007) David I Beaver, Brady Clark, Edward Stanton Flemming, T Florian Jaeger, and Maria Wolters. 2007. When semantics meets phonetics: Acoustical studies of second-occurrence focus. _Language_, 83(2):245-276.
* Bojanowski et al. (2017) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. _Transactions of the Association for Computational Linguistics_, 5:135-146.
* Cao and Lu (2017) Shaosheng Cao and Wei Lu. 2017. Improving word embeddings with convolutional feature learning and

\begin{table}
\begin{tabular}{c|l|l} \hline  **Vector** & **Embedding** & **Top 5C Guest Results** \\ \hline   \(E_{W}\) (\(\mathbb{H}\)) & \(E_{W}\) & \(\mathbb{H}\) (Zhao, \(\mathbb{K}\) (Una, \(\mathbb{K}\) (Dake Ding), \(\mathbb{R}\) (Chau), \(\mathbb{H}\) (Zhao Mao of Jin)) \\ \hline   \(E_{U}^{char}\) (\(\mathbb{H}\)) & \(E_{W}\) & \(\mathbb{H}\) (application), \(\mathbb{H}\) (Application), \(\mathbb{H}\) (Applicant), \\  & \(\mathbb{H}\) (Visu), \(\mathbb{H}\) (qualification), \(\mathbb{A}\) (entry), \\ \hline   \(E_{U}^{comp}\) & \(E_{W}\) & \(\mathbb{S}\) (disease), \(\mathbb{N}\) (infection), \\  & \(\mathbb{R}\) (plan), \(\mathbb{R}\) (lung), \(\mathbb{R}\) (systemptom) \\  & \(E_{U}^{char}\) & \(\mathbb{I}\) (symposium), \(\mathbb{I}\) (force), \(\mathbb{R}\) (ellipapsy), \\  & \(\mathbb{P}\) (subcern), \(\mathbb{I}\) (generate) \\ \hline   \(E_{Q}^{chop}\) t-ong & \(E_{W}\) & \(\mathbb{H}\) (pain), \(\mathbb{R}\) (learn), \(\mathbb{R}\) (anxiety), \\  & & \(\mathbb{H}\) (pain), \(\mathbb{R}\) (counters), \(\mathbb{R}\) (equpliate), \\  & \(E_{Q}^{char}\) & \(\mathbb{H}\) (pain), \(\mathbb{R}\) (increase) \\ \hline \end{tabular}
\end{table}
Table 5: Qualitative analysis. For a Vector, from an Embedding top 5 closest results are listed.



subword information. In _Thirty-First AAAI Conference on Artificial Intelligence_.
* Cao et al. (2018) Shaosheng Cao, Wei Lu, Jun Zhou, and Xiaolong Li. 2018. cw2vec: Learning chinese word embeddings with stroke n-gram information. In _Thirty-Second AAAI Conference on Artificial Intelligence_.
* Chen et al. (2015a) Heng Chen, Junying Liang, and Haitao Liu. 2015a. How does word length evolve in written chinese? _PloS one_, 10(9):e0138567.
* Chen et al. (2015b) Xinxiong Chen, Lei Xu, Zhiyuan Liu, Maosong Sun, and Huanbo Luan. 2015b. Joint learning of character and word embeddings. In _Twenty-Fourth International Joint Conference on Artificial Intelligence_.
* Collobert et al. (2011) Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. _Journal of machine learning research_, 12(Aug):2493-2537.
* Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. _arXiv preprint arXiv:1810.04805_.
* Kim et al. (2018) Yeachan Kim, Kang-Min Kim, Ji-Min Lee, and SangKeun Lee. 2018. Learning to generate word representations using subword information. In _Proceedings of the 27th International Conference on Computational Linguistics_, pages 2551-2561.
* Lafferty et al. (2001) John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data.
* Li et al. (2019) Xiaoya Li, Yuixian Meng, Xiaofei Sun, Qinghong Han, Arianna Yuan, and Jiwei Li. 2019. Is word segmentation necessary for deep learning of chinese representations? In _Proceedings of the 57th Conference of the Association for Computational Linguistics_, pages 3242-3252.
* Liu et al. (2017) Frederick Liu, Han Lu, Chieh Lo, and Graham Neubig. 2017. Learning character-level compositionality with visual features. _arXiv preprint arXiv:1704.04859_.
* Mikolov et al. (2013a) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. _arXiv preprint arXiv:1301.3781_.
* Mikolov et al. (2013b) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In _Advances in neural information processing systems_, pages 3111-3119.
* Mikolov et al. (2013c) Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013c. Linguistic regularities in continuous space word representations. In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 746-751.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_, pages 1532-1543.
* Peters et al. (2018) Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. _arXiv preprint arXiv:1802.05365_.
* Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. _Journal of Machine Learning Research_, 15(1):1929-1958.
* Sun et al. (2015) Yaming Sun, Lei Lin, Duyu Tang, Nan Yang, Zhenzhou Ji, and Xiaolong Wang. 2015. Modeling mention, context and entity with neural networks for entity disambiguation. In _Twenty-Fourth International Joint Conference on Artificial Intelligence_.
* Wieting et al. (2015) John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2015. Towards universal paraphastic sentence embeddings. _arXiv preprint arXiv:1511.08198_.
* Wieting et al. (2016) John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2016. Charagram: Embedding words and sentences via character n-grams. _arXiv preprint arXiv:1607.02789_.
* Yin et al. (2016) Rongchao Yin, Quan Wang, Peng Li, Rui Li, and Bin Wang. 2016. Multi-granularity chinese word embedding. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 981-986.
* Yu et al. (2017) Jinxing Yu, Xun Jian, Hao Xin, and Yangqiu Song. 2017. Joint embeddings of chinese words, characters, and fine-grained subcharacter components. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 286-291.