SlowGuess/ACL_2020_OCR huggingface dataset:
do not have the external sememe annotation during training and testing. Notably, ESD-sem also improves over Pip-sem by a large margin. This shows that the way our method leverages the sememe annotations, i.e. using them as external signals of word semantics, is more effective than simple annotate-then-generate pipeline methods.
In order to further compare the proposed methods and the strongest previous method (i.e., the Log-CaD model), we performed a human evaluation of the generated definitions. We randomly selected 100 samples from the test set of Oxford dataset, and invited four people with at least CET6 level English skills to rate the output definitions in terms of fluency and semantic completeness from 1 to 5 points. The averaged scores are presented in Table3. As can be seen from the table, definitions generated by our methods are rated higher in terms of semantic completeness while achieving comparable fluency.
We also perform an ablation study to quantify the effect of different model components.
Semantic completeness objective We can see that the semantic completeness objective, i.e.  $\mathcal{L}_{com}^{(*)}$  leads to a substantial improvement in terms of Meteor score (Line 3 and Line 4 vs. Line 1), which indicates that the gain obtained by our model is not by trivially adopting the conditional VAE framework to definition generation task.
Semantic diversity objective The experimental results show that although independently using the semantic diversity objective leads to no gains (Line 2 vs. Line 1), regularizing the model to learn diverse latent codes when using semantic completeness objective can improve the generation perfor
Lbase	Ldiv	L(def)com	L(sem)com	Meteor
1	✓	8.99
2	✓	✓	9.15
3	✓	✓	11.09
4	✓	✓	11.88
5	✓	✓	✓	11.56
6	✓	✓	✓	12.43
7	✓	✓	✓	✓	12.87
Table 4: Ablation study on the development set of Oxford dataset.
Figure 2: The Meteor scores of  $ESD$  on Oxford test dataset with different M and K, where M is the number of discrete latent variables used in  $ESD$ , and K is the number of categories.
mance of the model (Line 5 vs. Line 3 and Line 6 vs. Line 4).
To gain more insight into the improvement provided by the proposed method, we perform several analyses in this section.
5.1 Influence of the number of components
To validate that explicit decomposition of word semantics is beneficial for definition generation, we compare the performances of several models with different number of latent variables, and plot the result in Figure 2.
Overall, setting multiple latent variables given the same categories achieves noticeable improvements over  $M = 1$ , i.e. encoder-decoder model with word prediction mechanism. However, it is not the case we should adopt as many latent variables as possible. The reason for it is that generally a word has a limited number of semantic components (3-10 in HowNet), and having too many components in
Figure 3: Comparison between LOG-CaD and ESD-def with different parameter  $\delta$ .  $\delta$  controls how much we prefer content words over function words. Larger  $\delta$  implies we prefer content words more.
the latent models would damage the performance.
It is interesting to see that when we set the number of components  $\mathbf{M}$  to 8, the optimal number of categories  $\mathbf{K}$  is 256. As the total number of semantic units we are modeling is  $M \times K$ , this approximately equals to the number of sememes in HowNet.
5.2 Improvements on different word types
The goal of definition generation task is to accelerate dictionary compilation or to help humans with unfamiliar text. In both application scenarios, it is more important to generate content words that describe the semantic of the given word, rather than function words or phrases such as "refer to" and "of or relating to". To understand which kind of word our model achieves the largest improvements on, we evaluate Meteor scores of the baseline model and our model under different values of  $\delta$ , where  $\delta$  is a hyperparameter used by Meteor that controls how much we prefer content words over function words. Figure 3 shows the results. We can see that as our preference over content words increases, both the performances of baseline model and our model decreases, indicating that it is more difficult for current definition generation models to generate useful content words than function words. However, the gap between the baseline model and our model becomes larger when  $\delta$  increases, which shows that the gain of our model is mainly from the content words instead of function words.
Word
Context	militia
The militia repelled attacks from without and denied the executive the means to oppress from within.
Reference
LOG-CaD
ESD-def	a group of people who are not professional soldiers but who have had military training and can act as an army
a group of people engaged in a military force
a group of people engaged in a military force and not very skillful
Word
Context	captain
The captain gave the order to abandon ship
Reference
LOG-CaD
ESD-def	the person in charge of a ship
a person who is a member of ship
a person who is the leader of a ship
Table 5: Examples from LOG-CaD and ESD-def. We highlight the different part between two models in red.
word	z1	z2	z3	z4	z5	z6	z7	z8
red	54	7B	9C	60	A1	A7	F5	C7
yellow	54	92	7F	22	A1	A7	F5	55
blue	6A	E5	7F	22	A1	A7	F5	C7
cat	7A	E3	C4	22	A1	A7	F5	3B
dog	7A	43	C4	60	A1	A7	F5	3B
penguin	7A	C3	C4	60	A1	BE	F5	3B
Table 6: Examples of the learned latent codes. Each line is a word with the hexadecimal identifier of its corresponding latent codes. Color words like "red", "yellow", "blue" share most parts of latent codes with each other, while words from different groups like "red" and "cat" share fewer parts of latent codes.
5.3 Case Studies
Examples of learned latent codes In Table 6, we show some examples of learned latent codes on WordNet dataset. We can see that our model does learn informative codes, i.e. words with similar meanings are assigned with similar latent codes, and codes of words with different meanings tend to differ.
Examples of generated definitions We also list several generation samples in Table 5. We can see that the definitions generated by our method are more semantically complete than those by previous works, and they indeed capture fine-grained semantic components that the baseline model ignores. For example, it is necessary to know that militia has unprofessional military skills, which distinguishes the meaning of militia and army. The definition generated by the baseline model ignores this perspective. However, our model does describe the unprofessional nature of militia by generating "not very skillful", thanks to the ability of modeling
fine-grained semantic components.
Definition Generation Definition modeling was firstly proposed by Noraset et al. (2017). They take a word embedding as input and generate a definition of the word. An obvious drawback is that their model cannot handle polysemous words. Recently several works (Ni and Wang, 2017; Gadetsky et al., 2018; Ishiwatari et al., 2019) consider the context-aware definition generation task, where the context is introduced to disambiguate senses of words. They all adopt a encoder-decoder architecture, and rely heavily on the decoder to extract semantic components of the word semantic, thus leading to under-specific definitions. In contrast, we introduce a group of discrete latent variables to model these semantic components explicitly.
Semantic decomposition and Decomposed Semantics It is recognized by linguists that human beings understand complex meaning by decomposing it into components that are latent in the meaning. Wierzbicka (1996) propose that different languages share a set of atomic concepts that cannot be further decomposed i.e. semantic primitives, and all complex concepts can be semantically composed by semantic primitives. Dong and Dong (2003) introduce a similar idea. They call the atomic concepts as sememes, and present a knowledge base HowNet in which senses of words are annotated with sememes. HowNet is shown to be helpful for many NLP tasks, such as word representation learning (Niu et al., 2017), relation extraction (Li et al., 2019), aspect extraction (Luo
et al., 2019). Previously Yang et al. (2019) propose to use sememe annotations as a direct input when generating definitions, which can suffer from the data sparsity problem. In this paper, we instead leverage HowNet as the external supervising signals for latent variables when training and try to learn the knowledge into the model itself.
We proposed ESD, a context-aware definition generation model that explicitly models the decomposed semantics of words. Specifically, we model the decomposed semantics as discrete latent variables, and training with auxiliary losses to ensure that the model learns informative latent codes for definition modeling. As a result, ESD leads to significant improvements over the previous strong baselines on two established definition datasets. Quantitative and qualitative analysis showed that our model could generate more meaningful, specific and accurate definitions.
In future work, we plan to seek better ways to guide the learning of latent variables, such as using dynamic routing (Sabour et al., 2017) method to align the latent variables and sememes, and learn more explainable latent codes.
We would like to thank the anonymous reviewers for their insightful comments. Shujian Huang is the corresponding author. This work is supported by National Science Foundation of China (No. U1836221, 61772261), the Jiangsu Provincial Research Foundation for Basic Research (No. BK20170074).
Yu Bao, Hao Zhou, Shujian Huang, Lei Li, Lili Mou, Olga Vechtomova, Xinyu Dai, and Jiajun Chen. 2019. Generating sentences from disentangled syntactic and semantic spaces. In ACL, pages 6008-6019.
Leonard Bloomfield. 1949. A set of postulates for the science of language. *IJAL*, 15(4):195–202.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. TACL, 5:135-146.
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. In CoNLL, pages 10-21.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP, pages 1724-1734.
Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the EACL 2014 Workshop on Statistical Machine Translation.
Zhendong Dong and Qiang Dong. 2003. HowNet - a hybrid language and knowledge resource. In NLPKE, pages 820-824.
Carol Fraser. 1999. The role of consulting a dictionary in reading and vocabulary learning. Canadian Journal of Applied Linguistics, 2:73-89.
Artyom Gadetsky, Ilya Yakubovskiy, and Dmitry Vetrov. 2018. Conditional generators of words definitions. In ACL, pages 266-271.
Cliff Goddard and Anna Wierzbicka. 1994. Semantic and Lexical Universals: Theory and Empirical Findings, volume 25. John Benjamins Publishing.
Shonosuke Ishiwatari, Hiroaki Hayashi, Naoki Yoshi-naga, Graham Neubig, Shoetsu Sato, Masashi Toyoda, and Masaru Kitsuregawa. 2019. Learning to describe unknown phrases with local and global contexts. In NAACL, pages 3467-3476.
Vineet John, Lili Mou, Hareesh Bahuleyan, and Olga Vechtomova. 2019. Disentangled representation learning for text style transfer. In ACL.
Lukasz Kaiser, Samy Bengio, Aurko Roy, Ashish Vaswani, Niki Parmar, Jakob Uszkoreit, and Noam Shazeer. 2018. Fast decoding in sequence models using discrete latent variables. In ICML, pages 2395-2404.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In NIPS, pages 1097-1105.
Ziran Li, Ning Ding, Zhiyuan Liu, Haitao Zheng, and Ying Shen. 2019. Chinese relation extraction with multi-grained information and external linguistic knowledge. In ACL, pages 4377-4386.
Ling Luo, Xiang Ao, Yan Song, Jinyao Li, Xiaopeng Yang, Qing He, and Dong Yu. 2019. Unsupervised neural aspect extraction with sememes. In *IJCAI*, pages 5123-5129.
Ke Ni and William Yang Wang. 2017. Learning to explain non-standard english words and phrases. CoRR, abs/1709.09254.
Yilin Niu, Ruobing Xie, Zhiyuan Liu, and Maosong Sun. 2017. Improved word representation learning with sememes. In ACL, pages 2049-2058.
Thanapon Noraset, Chen Liang, Larry Birnbaum, and Doug Downey. 2017. Definition modeling: Learning to define word embeddings in natural language. In AAAI.
Aaron van den Oord, Oriol Vinyals, et al. 2017. Neural discrete representation learning. In Advances in Neural Information Processing Systems, pages 6306-6315.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In ACL, pages 311-318.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In EMNLP, pages 1532-1543.
Aurko Roy, Ashish Vaswani, Niki Parmar, and Arvind Neelakantan. 2018. Towards a better understanding of vector quantized autoencoders. arXiv.
Sara Sabour, Nicholas Frosst, and Geoffrey E. Hinton. 2017. Dynamic routing between capsules. CoRR, abs/1710.09829.
Raphael Shu, Jason Lee, Hideki Nakayama, and Kyunghyun Cho. 2019. Latent-variable non-autoregressive neural machine translation with deterministic inference using a delta posterior. In AAAI.
Martin Sundermeyer, Ralf Schluter, and Hermann Ney. 2012. Lstm neural networks for language modeling. In InterSpeech.
Koki Washio, Satoshi Sekine, and Tsuneaki Kato. 2019. Bridging the defined and the defining: Exploiting implicit lexical semantic relations in definition modeling. In EMNLP-IJCNLP, pages 3519-3525.
Rongxiang Weng, Shujian Huang, Zaixiang Zheng, XIN-YU DAI, and CHEN Jiajun. 2017. Neural machine translation with word predictions. In EMNLP, pages 136-145.
Anna Wierzbicka. 1996. Semantics: Primes and Universals. Oxford University Press.
Ruobing Xie, Xingchi Yuan, Zhiyuan Liu, and Maosong Sun. 2017. Lexical sememe prediction via word embeddings and matrix factorization. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pages 4200-4206. AAAI Press.
Liner Yang, Cunliang Kong, Yun Chen, Yang Liu, Qinan Fan, and Erhong Yang. 2019. Incorporating sememes into chinese definition modeling. arXiv preprint arXiv:1905.06512.
Biao Zhang, Deyi Xiong, Jinsong Su, Hong Duan, and Min Zhang. 2016. Variational neural machine translation. In EMNLP.