SlowGuess/NAACL_2021_OCR huggingface dataset:
Fine-Grained Pipeline Fig.4 verifies the fine-grained pipeline's effectiveness. Given fields and loss function of the model, leveraging the whole fine-grained pipeline including n-gram and grain dropping can lead to advanced word embeddings. For example, in the W.H.C model, the fine-grained pipeline yields a  $9.13\%$  increase in the Chinese Analogy task. Beyond this, Fig. 5 further validates the effectiveness of grain dropping. In the W.H.P model, grain dropping achieves a  $1.57\%$  improvements in Analogy task. The improvements demonstrate that our fine-grained pipeline does produce high-quality grain sequences and can successfully capture more linguistic information that previous works missed.
Loss Function Fig. 6 asserts the effectiveness of our loss function.  $L_{fe}$  loss function outperforms other loss functions in all tasks. For example, in Sim240 task,  $L_{fe}$  achieves  $4.49\%$  increase compared to  $L_{w2v}$  and  $3.04\%$  increase compared to  $L_{jwe}$ . It's the same for Sim297 and Analogy. Compared with  $L_{w2v}$  and  $L_{jwe}$ ,  $L_{fe}$  considers prominent linguistic phenomena such as morpheme-phoneme pairs. Therefore, it successfully captures the within-field and crossing-field information.
5.2 Downstream Task
We conduct both Chinese and English downstream tasks, including text classification and NER, to test the performances of word embeddings and grain embeddings.
Text Classification Table 5.2 shows with more linguistic fields, both word and grain embeddings gain performance improvements on text classification tasks. This reveals that phoneme and syntax fields can improve the quality of both word and grain embeddings. Moreover, grain embeddings  $E_{G}$  always outperform word embeddings  $E_{W}$ . In NewsGroup task,  $E_{G}$  of W.C.P-pos exceed  $E_{W}$  with  $3.08\%$ . Furthermore, compared with word embeddings, grain embeddings make larger improvements with more fields. For exam-
Figure 4: Fine-grained pipeline performances on lexical tasks.
Figure 5: Grain dropping performances on lexical tasks.
Figure 6: Loss function performances on lexical tasks.
Method	CoNLL2003 (English)	Boson (Chinese)
EW	EG	EW	EG
Word2Vec	77.04	-	64.13	-
W.C	77.19	80.35	64.96	65.08
W.C.P	78.06	82.01	65.08	65.34
W.C.PPos	78.44	86.92	65.40	71.75
ple, from W.C to W.C.PPos,  $E_W$  achieve a  $4.06\%$  increase while  $E_G$  achieves a  $4.76\%$  increase in NewsGroup task. Such additional improvement derives from additional linguistic information that is not included in word embeddings. This strongly indicates that grain embeddings carry more linguistic information than the associated word embeddings and can be a better alternative to word embeddings.
Named Entity Recognition As to NER performances of Table 3, a similar pattern is observed to that of text classification. It demonstrates that more linguistic fields benefit the NER tasks. For example, from W.C.P to W.C.P(Pos, grain embeddings gain  $4.91\%$  and  $6.41\%$  improvements in CoNLL2003 and Boson tasks. The reason for the improvements is that the hyperfield POS carries the part of a sentence's syntactical information, which is crucial in sequence labeling tasks. This illustrates the hyperfield's significance in learning embeddings. Moreover, it shows grain embeddings outperform word embeddings. For instance, in W.C.P(Pos model, F1 scores of grain embeddings exceed that of word embeddings with  $8.48\%$  and  $6.35\%$  in CoNLL2003 and Boson tasks.
To further prove that our embeddings are effective with complicated neural networks, we adopt Embed-BiLSTM-CRF and conduct experiments on CoNLL2003. Besides static embedding Word2Vec, dynamic embedding methods, such as ELMo and BERT, are also listed as baselines for
Table 3: Comparison of the proposed method with the Embed-CRF structure and Word2Vec on NER. F1 score is presented in percentage  $(\%)$ .
Method	F1 Score (%)
W.C	91.82
W.C.P	92.16
W.C.P(Pos)	92.34
Word2Vec (Mikolov et al., 2013b)	90.72
ELMo (Peters et al., 2018)	92.22
BERT (Devlin et al., 2018)	92.80
Table 4: Different embeddings with BiLSTM-CRF structure on the CoNLL2003 NER task.
comparison, as shown in Table 4. In terms of static embeddings, additional fields still benefit the task, with around  $0.54\%$  increase per field from Word2Vec to W.C.P(Pos. This shows that, even in complex neural networks, grain embeddings are superior to word embeddings, and phoneme and syntax are useful.
For dynamic embeddings, though it is marginally inferior to BERT, our W.C.PPos is better than EMLo by  $0.12\%$ . It indicates our framework exploits the potential of static embeddings with multiple fields which surpasses the relatively shallow dynamic embeddings. This also suggests the potential of introducing multi-fields to dynamic embeddings. Our static embeddings also enjoy other advantages. The model's structure is simple and straightforward and parameter size is small. It requires less corpus and resources to train compared to BERT, which is a complicated deep neural networks. In downstream tasks, though BERT outperforms our model, it bears expensive costs of model complexity and computational resources. Moreover, the dynamic embeddings cannot represent the independent word or gain, whereas our model yields high quality representations for them and achieves best performances in lexical tasks.
5.3 Qualitative Evaluation
We evaluate the embeddings' abilities to uncover the semantic relatedness of words, characters, and components through case studies based on a model trained with character, component, and pinyin. Taking 申 (apply) as an example, which is also an ancient state name and a popular last name in China, it can be a Chinese character or word. We list its closest words from word embeddings in Table 5 where we treat 申 as a character and a word. When it is a character, most of the closest words are semantically related to apply. When it acts as a word, the closest words related to country name and last name meanings in 申. For example, 赵 (Zhao), 殷 (Yin) are ancient state names and last names, and 定公 (Duke Ding) 晋昭公 (Duke Zhao of Jin) are dukes in ancient China. This reveals that grain embeddings can supplement the word embeddings for a more complete semantic representation.
We further take the component (illness) as an example and Table 5 shows its closest characters and words. All of the closest characters and words are semantically related to the component. Most of them are related to diseases, symptoms and other medical terms, such as words疾病 (disease),感染(infection)and characters症 (symptom),疮(score).Most of them contain component, but 肿(gangrene),患(suffer),and感染(infection)without沪 also share the similar semantics. Moreover, we study pinyin t-ong,sound of痛(pain),and list its closest words and characters in Table 5 and observe a similar phenomenon to。 These closest characters have similar semantic meaning with 痛(pain),whose pinyin is t-ong,such as words疼痛(pain),头痛(headache), and characters疮(sore)，瘫(paralysis).The qualitative analysis shows that our proposed models leverage both external context co-occurrence information and internal morphological and phonetic information. The medical information stored in above grain embeddings could be utilized for clinical NER tasks.
We propose a flexible field embedding framework to jointly learn both word and grain embeddings by incorporating morphological, phonetic, and syntactical linguistic fields simultaneously. Our proposed framework leverages an innovative fine-grained pipeline, including n-grams and grain dropping, as well as a novel loss function to cap
Vector	Embedding	Top 5 Closest Results
EW申	EW	赵(Zhao),殷(Yin),定公(Duke Ding),楚(Chu),晋昭公(Duke Zhao of Jin)
EGchar申	EW	申请(application),申请人(applicant),签证(visa),资格(qualification),入境(entry)
EGcomp扩	EW	疾病(disease),感染(infection),疼痛(pain),肺(lung),症状(symptom)
EGchar	症(symptom),疮(sore),癫(epilepsy),肿(swollen),疽(gangrene)
EGpinyin t-ong	EW	痛(pain),流泪(tear),焦虑(anxiety),痛楚(pain),呕(vomit)
EGchar	痛(pain),疣(remorse),悸(palpitate),症(disease),嚏(sneeze)
Table 5: Qualitative analysis. For a Vector, from an Embedding top 5 closest results are listed.
ture the information contained within each linguistic field and shared across multiple fields. By introducing phonetic and syntactical linguistic fields and leveraging our fine-grained pipeline and loss function, our framework is capable of learning better word embeddings in terms of word similarity and analogy. Furthermore, we systemically investigate the effectiveness of grain embeddings and provides the evidence that grain embeddings can be a better alternative to word embeddings for word representations. Experimental results show that grain embeddings outperform word embeddings in several downstream NLP tasks, such as text classification and named entity recognition. The qualitative analysis illustrates that grain embeddings can effectively capture semantic information.
We would like to thank Prof. Jordan Boyd-Graber from University of Maryland and Dr. Yefeng Zheng from Tencent for their helpful suggestions. This work was supported by the Key-Area Research and Development Program of Guangdong Province [2020B0101350001].
David I Beaver, Brady Clark, Edward Stanton Flemming, T Florian Jaeger, and Maria Wolters. 2007. When semantics meets phonetics: Acoustical studies of second-occurrence focus. Language, 83(2):245-276.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146.
Shaosheng Cao and Wei Lu. 2017. Improving word embeddings with convolutional feature learning and
subword information. In Thirty-First AAAI Conference on Artificial Intelligence.
Shaosheng Cao, Wei Lu, Jun Zhou, and Xiaolong Li. 2018. cw2vec: Learning chinese word embeddings with stroke n-gram information. In Thirty-Second AAAI Conference on Artificial Intelligence.
Heng Chen, Junying Liang, and Haitao Liu. 2015a. How does word length evolve in written chinese? PloS one, 10(9):e0138567.
Xinxiong Chen, Lei Xu, Zhiyuan Liu, Maosong Sun, and Huanbo Luan. 2015b. Joint learning of character and word embeddings. In Twenty-Fourth International Joint Conference on Artificial Intelligence.
Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of machine learning research, 12(Aug):2493-2537.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Yeachan Kim, Kang-Min Kim, Ji-Min Lee, and SangKeun Lee. 2018. Learning to generate word representations using subword information. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2551-2561.
John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data.
Xiaoya Li, Yuxian Meng, Xiaofei Sun, Qinghong Han, Arianna Yuan, and Jiwei Li. 2019. Is word segmentation necessary for deep learning of Chinese representations? In Proceedings of the 57th Conference of the Association for Computational Linguistics, pages 3242-3252.
Frederick Liu, Han Lu, Chieh Lo, and Graham Neubig. 2017. Learning character-level compositionality with visual features. arXiv preprint arXiv:1704.04859.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111-3119.
Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013c. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the
Association for Computational Linguistics: Human Language Technologies, pages 746-751.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929-1958.
Yaming Sun, Lei Lin, Duyu Tang, Nan Yang, Zhenzhou Ji, and Xiaolong Wang. 2015. Modeling mention, context and entity with neural networks for entity disambiguation. In Twenty-Fourth International Joint Conference on Artificial Intelligence.
John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2015. Towards universal paraphrastic sentence embeddings. arXiv preprint arXiv:1511.08198.
John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2016. Charagram: Embedding words and sentences via character n-grams. arXiv preprint arXiv:1607.02789.
Rongchao Yin, Quan Wang, Peng Li, Rui Li, and Bin Wang. 2016. Multi-granularity Chinese word embedding. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 981-986.
Jinxing Yu, Xun Jian, Hao Xin, and Yangqiu Song. 2017. Joint embeddings of chinese words, characters, and fine-grained subcharacter components. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 286-291.