Venue: NAACL 2021

# Decompose, Fuse and Generate:

A Formation-Informed Method for Chinese Definition Generation

Hua Zheng1,2, Damai Dai1,2 , Lei Li1,2, Tianyu Liu1,2,

**Zhifang Sui1,2,3, Baobao Chang1,2,3, Yang Liu1,2\({}^{1}\)Department of Computer Science and Technology, Peking University \({}^{2}\)Key Lab of Computational Linguistics (MOE), Peking University \({}^{3}\)Peng Cheng Laboratory, China {zhenghua,daidamai,tianyu0421,szf,chbb,liuyang}@pku.edu.cn lilei@stu.pku.edu.cn**

Equal contribution.Corresponding author.The code is available at [https://github.com/](https://github.com/) Hunter-DDM/DeFT-naac12021.

###### Abstract

In this paper, we tackle the task of Definition Generation (DG) in Chinese, which aims at automatically generating a definition for a word. Most existing methods take the source word as an indecomposable semantic unit. However, in partaxis languages like Chinese, word meanings can be composed using the word formation process, where a word ("\(\nexistsuit\)", _peach-blossom_) is formed by formation components ("\(\nexistsuit\)", _peach_; "\(\negmedurkey\)", _flower_) using a formation rule (_Modifier-Head_). Inspired by this process, we propose to enhance DG with word formation features. We build a formation-informed dataset and propose a model **DeFT**, which decomposes words into formation features, dynamically Fuses different features through a gating mechanism, and generaTes word definitions. Experimental results show that our method is both effective and robust.1

Footnote 1: Corresponding author.

+
Footnote †: The code is available at [https://github.com/](https://github.com/) Hunter-DDM/DeFT-naac12021.

+
Footnote †: The code is available at [https://github.com/](https://github.com/) Hunter-DDM/DeFT-naac12021.

## 1 Introduction

Definition Generation (DG) aims at automatically generating an explanatory text for a word. This task is of practical importance to assist dictionary construction, especially in highly productive languages like Chinese [5]. Most existing methods take the source word as an indecomposable lexico-semantic unit, using features like word embedding [8] and context [6, 7]. Recently, yang2020learning and li2020learning achieve improvement by decomposing the word meaning into different semantic components.

In decomposing the word meaning, the word formation process is an intuitive and informative way that has not been explored in DG by far. For parataxis languages like Chinese, a word is formed by formation components, i.e., **morphemes**, and a formation **rule**. As shown in Figure 1, the polysemous word "\(\exists\)" holds two meanings "\(\exists\)" and "\(\exists\)"\(\forall\)", which can be distinguished by different morphemes ("\(\exists_{1}\)"\(\forall\)" vs. "\(\exists_{2}\)"\(\forall\)") and different rules (_Modifier-Head_ vs. _Adverb-Verb_). Such intuitive formation process can clearly and unambiguously construct the word meaning.

Inspired by the word formation process in Chinese, we propose to enhance DG with formation features. First, we build a formation-informed dataset under expert annotations. Next, we design a DG model **DeFT**, which Decomposes words into formation features, Fuses different features through a gating mechanism, and generaTes definitions.

Our contributions are as follows: (1) We first propose to use word formation features to enhance DG and design a formation-informed model **DeFT**. (2) We build a new formation-informed DG dataset under expert annotations. (3) Experimental results show that our method brings a substantial performance improvement, and maintains a robust performance even with only word formation features.

## 2 Related Work

**Definition Generation:** Noraset et al. (2017) first propose the DG task and use word embeddings as the main input. The following methods add contexts for disambiguation [9, 10] or word-pair embeddings to capture lexical relations [11].

Figure 1: Word formation process for the polysemous ”\(\exists\)”\(\exists\)”. With morphemes and a formation rule specified, the process can construct and distinguish each meaning. Definitions are simplified for ease of reading.



Recent methods attempt to decompose the word meaning by using HowNet sememes Yang et al. (2020) or modeling latent variables Li et al. (2020).

**Semantic Components:** To systematically define words, linguists decompose the word meaning into semantic components Wierzbicka (1996). Following this idea, HowNet Dong and Dong (2006) uses manually-created semenes to describe the semantic aspects of words. Recent studies also show that leveraging subword information produces better embeddings Park et al. (2018); Lin and Liu (2019); Zhu et al. (2019), but these methods lack a clear distinction among different formation rules.

## 3 Word Formation Process in Chinese

It is linguistically motivated to explore the word formation process to better understand words. Instead of combining roots and affixes, Chinese words are formed by characters in a parataxis way Li et al. (2018). Here, we introduce two formation features and construct a formation-informed dataset.

### Formation components and rules

Chinese formation components are **morphemes**, defined as the smallest meaning-bearing units Zhu (1982). Morphemes are unambiguous in representing word meanings, since they can distinguish different meanings and uses of each character in a word, like "\(\boxed{\mathbb{E}_{1}}\)" and "\(\boxed{\mathbb{E}_{2}}\)" in Figure 1. Morphemes are also productive in constructing words, since over 99.48% Chinese words are formed using a small set of nearly 20,000 morphemes Fu (1988). These properties make morphemes highly effective as formation components.

Formation **rules** specify how morphemes are combined to form words in a partaraxis way. For example, the _Modifier-Head_ rule uses the first morpheme to modify the second morpheme. Following the study of Liu et al. (2018), we adopt 16 Chinese formation rules and show the top 5 in instance percentage in Table 1. Complete descriptions of 16 formation rules are provided in Appendix A.

### Formation-informed dataset

We construct a DG dataset under expert annotations, which contains morphemes and formation rules. Each entry consists of (1) source word, (2) morphemes and morpheme definitions, (3) formation rule, (4) context (a sentence containing the source word), (5) source word definition.

To ensure full coverage and fine granularity, we extract data from the 5th edition of the Contemporary Chinese Dictionary published by the Commercial Press2, one of the most influential Chinese dictionaries. We collect 45,311 Chinese disyllabic word entries with contexts and definitions. To annotate them, we also collect 10,527 Chinese characters and 20,855 morphemes with definitions.

Footnote 2: [https://www.cp.com.cn](https://www.cp.com.cn)

Our annotators include two professors and six graduates major in Chinese linguistics. Given the definition, they annotate each word with its formation rule (as shown in Table 1) and morpheme IDs (as shown in Table 2). Each entry is cross-validated by three independent annotators and reviewed by one. The detailed annotation process includes the following three steps:

* Equipped with the definition, annotators annotate each entry with two morpheme IDs (select from the morphemes of each character) and a formation rule (select from 16 formation rules). Each entry is independently annotated by three annotators, who also note down a confidence score. If three annotations are the same, turn to (3); otherwise, turn to (2).
* Another annotator reviews the conflicting annotations and confidence scores, and decides the final annotation. Turn to (3).
* The annotation is collected as an entry into the final dataset.

It takes one minute on average for each annotator to annotate an entry. Only 8,193 out of 45,311 entries enter Phase (2) in the whole process.

\begin{table}
\begin{tabular}{l|c|c} \hline \hline
**Formation Rule** & **Use Case** & **\%** \\ \hline Modifier-Head & \(\sharp\)L\% (red-flower) & 38.62 \\ Parallel & \(\sharp\)L\% (dizzy-dim) & 22.87 \\ Verb-Object & \(\sharp\)L\% (spend-money) & 16.44 \\ Adverb-Verb & \(\sharp\)L\% (vainly-spend) & 8.45 \\ Single Morpheme & \(\sharp\)L\% (peanut) & 3.51 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Examples of word formation rules and use cases. % denotes the instance percentage.

\begin{table}
\begin{tabular}{l|c} \hline \hline
**Morpheme (ID)** & **Morpheme Definition** \\ \hline \(\ddagger\)L\textbackslash{} (07361-01) & \(\sharp\)L\% (flower) \\ \(\ddagger\)L\textbackslash{} (07361-06) & \(\sharp\)L\% (dim; blurred) \\ \(\ddagger\)L\textbackslash{} (07361-09) & \(\sharp\)L\% (use; spend) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Three example morphemes and definitions for the character “\(\ddagger\)”. We give each morpheme a unique ID, C-M (C is character rank, M is morpheme rank).

 



## 4 Approach

### Task formulation

We extend the DG setting in Ishiwatari et al. (2019) to incorporate the word formation features, \(F=\{morph_{1},morph_{2},rule\}\), where \(morph_{i}\) is the \(i^{\text{th}}\) morpheme definition sentence and \(rule\) is the formation rule. The training goal is to maximize the likelihood of the ground-truth definition \(D=d_{1:T}\) given the source word \(w_{*}\), the context sentence \(C=c_{1:n}\), and the word formation features \(F\):

\[p(D|w_{*},C,F)=\prod_{t=1}^{T}p(d_{t}|d_{i<t},w_{*},C,F).\]

Our optimization objective is to minimize the cross-entropy loss \(\mathcal{L}\):

\[\mathcal{L}=-\sum_{t=1}^{T}\log\big{(}p(d_{t}|d_{i<t},w_{*},C,F)\big{)},\]

where \(d_{1:T}\) is the ground-truth definition, \(w_{*}\) is the pretrained embedding of the source word, \(C\) is the context sentence, \(F\) is the formation information.

### Proposed model: DeFT

As shown in Figure 2, DeFT first produces a seed vector in a rule-specific manner as global supervision. Then we feed it into the definition generator, which uses a gating mechanism to dynamically fuse different features and generate definitions.

#### 4.2.1 Seed vector

We first employ a Bi-LSTM Graves and Schmidhuber (2005) to encode \(morph_{i}\). Then, we combine \(mroph_{i}\) into a comprehensive morpheme embedding \(\mathbf{r}_{m}\) with a rule-specific linear layer, which captures different semantic relations:

\[\mathbf{m}_{i} =\text{Bi-LSTM}([morph_{i}]),\] \[\mathbf{r}_{m} =\mathbf{W}_{m}^{(rule)}[\mathbf{m}_{1};\mathbf{m}_{2}]+\mathbf{ b}_{m}^{(rule)}.\]

We then use a linear layer to combine \(\mathbf{r}_{m}\) and the pretrained source word embedding \(\mathbf{w}_{*}\) to obtain the seed vector \(\mathbf{r}_{*}\) as the initial generator input:

\[\mathbf{r}_{*}=\mathbf{W}_{r}[\mathbf{r}_{m};\mathbf{w}_{*}]+\mathbf{b}_{r}.\]

#### 4.2.2 Definition generator

We employ an LSTM followed by a GRU-like Cho et al. (2014) gate GRU-GATE\((\cdot)\), which dynamically fuses different features, as the generator:

\[\mathbf{h}_{t} =\text{LSTM}(\mathbf{d}_{t-1},\mathbf{h}_{t-1}^{\prime}),\] \[\mathbf{h}_{t}^{\prime} =\text{GRU-GATE}(\mathbf{h}_{t},\textbf{feat}_{t}),\] \[\textbf{feat}_{t} =[\mathbf{r}_{m};\mathbf{w}_{*};\mathbf{a}_{*};\mathbf{g}_{t}; \mathbf{c}_{t}],\]

where \(\mathbf{h}_{t}\) is the LSTM hidden state at the \(t^{\text{th}}\) step, \(\mathbf{h}_{t}^{\prime}\) is the gated hidden state, \(\mathbf{d}_{t-1}\) is the embedding of the previous definition word, specially, \(\mathbf{d}_{0}\triangleq r_{*}\), and \(\textbf{feat}_{t}\) denotes the features that dynamically control the generation process. We explain \(\mathbf{a}_{*}\), \(\mathbf{g}_{t}\), and \(\mathbf{c}_{t}\) as follows.

\(\mathbf{a}_{*}\) is the character-level embedding, obtained by combining the embedding \(\mathbf{ch}_{i}\) of each character in \(w_{*}\) with a rule-specific linear layer:

\[\mathbf{a}_{*}=\mathbf{W}_{a}^{(rule)}[\mathbf{ch}_{1};\mathbf{ ch}_{2}]+\mathbf{b}_{a}^{(rule)}.\]

\(\mathbf{g}_{t}\) is the gated attended morpheme vector that dynamically focuses on the most relevant parts in morphemes during generation. We first calculate attended morpheme vectors \(\mathbf{g}_{t,i}^{\prime}\) by the attention mechanism Bahdanau et al. (2015):

\[\mathbf{g}_{t,i}^{\prime}=\text{Attention}(\mathbf{h}_{t},morph_{ i}),\]

where \(\text{Attention}(\mathbf{h},seq)\) denotes the function that uses \(\mathbf{h}\) to attend over the Bi-LSTM encoded \(seq\). We then design a **MorphGATE** to compute \(\mathbf{g}_{t}\) by assigning different weights to two morphemes:

\[\mathbf{z}_{t} =\sigma(\mathbf{W}_{z}[\mathbf{g}_{t,1}^{\prime};\mathbf{g}_{t,2} ^{\prime};\mathbf{h}_{t}]+\mathbf{b}_{z}),\] \[\mathbf{g}_{t} =(\mathbf{1}-\mathbf{z}_{t})\odot\mathbf{g}_{t,1}^{\prime}+ \mathbf{z}_{t}\odot\mathbf{g}_{t,2}^{\prime},\]

Figure 2: An illustration of DeFT. The definition generator uses the seed vector as input, and dynamically fuses different features using a gating mechanism.



where \(\sigma(\cdot)\) is Sigmoid and \(\odot\) is Hadamard product.

\({\bf c}_{t}\) is the attended context vector. Following Ishiwatari et al. (2019), we take \({\bf c}_{t}=\text{Attention}({\bf h}_{t},C)\) as a feature since it may assist disambiguation.

Finally, GRU-GATE\(({\bf h}_{t},\mathbf{feat}_{t})\) takes the LSTM hidden state \({\bf h}_{t}\) and the dynamically controlled features \(\mathbf{feat}_{t}\) as input, and updates \({\bf h}_{t}\) to \({\bf h}^{\prime}_{t}\) by fusing different features:

\[{\bf u}_{t} =\sigma({\bf W}_{u}[{\bf h}_{t};\mathbf{feat}_{t}]+{\bf b}_{u}),\] \[{\bf v}_{t} =\sigma({\bf W}_{r}[{\bf h}_{t};\mathbf{feat}_{t}]+{\bf b}_{r}),\] \[{\bf\hat{h}}_{t} =\text{tanh}({\bf W}_{h}[({\bf v}_{t}\odot\mathbf{feat}_{t});{\bf h }_{t}]+{\bf b}_{h}),\] \[{\bf h}^{\prime}_{t} ={\bf u}_{t}\odot{\bf h}_{t}+({\bf 1}-{\bf u}_{t})\odot{\bf\hat{h}}_{ t},\]

where \(\sigma\) denotes the Sigmoid and \(\odot\) denotes the Hadamard product. The gate \({\bf u}_{t}\) controls how much the original state \({\bf h}_{t}\) is remained, and the gate \({\bf v}_{t}\) controls the contribution from features \(\mathbf{feat}_{t}\).

## 5 Experiments

### Experimental settings

**Dataset:** We split the dataset described in Section 3 into training, validation and test sets by 8:1:1, as shown in Table 3. Note that we treat polysemous words as different entries, and the words are mutually exclusive across three sets.

**Hyper-parameters:** We tune hyper-parameters to achieve the best BLEU score on the validation set. We use Adam Kingma and Ba (2015) with an initial learning rate of \(10^{-3}\) as the optimizer. We set hidden size to 300, batch size to 64 and dropout rate to 0.2. Word embeddings are 300-dimensional, pretrained by fastText Bojanowski et al. (2017). We train for up to \(50\) epochs, and early stop the training process once the performance does not improve for \(10\) consecutive epochs. We run our experiments on a single NVIDIA GeForce GTX 2080Ti GPU with 11 GB memory.

**Baselines:** We compare with two reproducible baselines that have a similar model framework with us but using different features, including SG Norsset et al. (2017) that uses only the word feature, and LOG-CaD Ishiwatari et al. (2019) that uses both the word and context features.

### Evaluation results

We conduct both automatic and human evaluations to validate our method, and show results in Table 4.

For automatic evaluation, we select BLEU-4 Bahdanau et al. (2015) and ROUGE-L Lin (2004) as metrics. We find that (1) our formation-informed DeFT (F+W+C) significantly outperforms baselines and other simplified versions (F, F+W, F+C); (2) based on W or W+C, adding formation features introduces significant improvement; (3) formation features are robust, since using only F can outperform LOG-CaD by 9.8% and 10.45% in BLEU and ROUGE-L, respectively. These findings validate that formation features can effectively enhance DG by assisting word meaning construction.

For human evaluation, we measure semantic **coverage** and **overall** quality. The **coverage** metric measures how much ground-truth information is mentioned in the predicted definition. To be specific, the scores are given based on how many semantic aspects in the ground truth definition are covered by the predicted definition. The **overall** metric measures the overall quality of the predicted definition, referencing the ground-truth definition. We randomly select 100 entries from the test set, and hire three raters to rate the predicted definitions on a scale of 1 to 5, where each entry includes (1) the source word, (2) the ground-truth definition, and (3) the predicted definition to the raters. We show in Table 5 the detailed guideline for raters on each point.

The inter-rater kappa Fleiss and Cohen (1973) is 0.65 for coverage and 0.66 for overall. We average scores of raters and obtain consistent results with

\begin{table}
\begin{tabular}{l|c c|c c} \hline  & \multicolumn{2}{c|}{**Automatic**} & \multicolumn{2}{c}{**Human**} \\  & **BLEU** & **ROUGE-L** & **Coverage****Overall** \\ \hline W (SG*) & 23.63 & 34.77 & 2.04 & 2.11 \\ W+C (LOG-CaD*) & 24.06 & 34.93 & 2.52 & 2.45 \\ \hline F & 25.55 & 37.27 & 2.61 & 2.58 \\ F+W & 25.93 & 37.94 & 2.78 & 2.71 \\ F+C & 25.69 & 37.29 & 2.89 & 2.78 \\ F+W+C (DeFT) & **26.42** & **38.58** & **3.29** & **3.19** \\ \hline \end{tabular}
\end{table}
Table 4: Automatic and human evaluation results. Best results are in **Bold**. W, C and F are word, context and formation features, respectively. * denotes baselines.

\begin{table}
\begin{tabular}{l|c c c c c} \hline \multirow{2}{*}{**Split**} & \#**Works \#Entries** & \begin{tabular}{c} Context Morph\({}_{i}\) \\ Length \\ \end{tabular} & \begin{tabular}{c} Morph\({}_{j}\) \\ Length \\ \end{tabular} & \begin{tabular}{c} Definition \\ Length \\ \end{tabular} & \begin{tabular}{c} Length \\ \end{tabular} & 
\begin{tabular}{c} Length \\ \end{tabular} \\ \hline Train & 29,169 & 36,248 & 7.22 & 7.69 & 7.29 & 12.02 \\ Valid & 3,673 & 4,531 & 7.32 & 7.45 & 7.30 & 11.91 \\ Test & 3,666 & 4,532 & 7.26 & 7.51 & 7.01 & 12.03 \\ \hline \end{tabular}
\end{table}
Table 3: Statistics of our formation-informed dataset. Morph\({}_{i}\) denotes the definition of the \(i^{\text{th}}\) morpheme. The length is calculated as the average number of Chinese characters.

 

the automatic evaluation: formation features are effective and DeFT performs the best.

### Analysis

**Ablation study:** Based on DeFT, we perform ablation study regarding MorphGATE and the formation rule in Table 6. (1) For MorphGATE, we replace it with a simple average function, which leads to a drop in performance. This reveals that different morphemes take effect in different generation phases. (2) For formation rule, we replace the rule-specific layers with a rule-shared layer, leading to a more serious performance drop. This verifies that distinguishing the specific formation rule can assist word meaning construction.

**Formation features can assist disambiguation:** We present the generated definitions for a polysemous word in Figure 3. The example shows that using only the word feature (W) cannot distinguish different meanings. By contrast, using only the formation features (F) can capture the meaning difference and disambiguate the word (_use_ vs. _money_). Further, DeFT (F+W+C) generates the exactly correct definition by fusing different features. Due to space limits, we put two additional interesting analyses on formation rules in Appendix B.

**Formation features are more feasible and effective compared with sememes:** Sememes are expert-crafted words to describe the semantic aspects of words. For annotation cost, annotating sememes is as expensive as writing definitions Li et al. (2020), whereas annotating formation features is a simple multiple-choice task with 1.98 choices on average. For effectiveness, we conduct experiments using sememe embeddings from Yang et al. (2020) as additional features. Results show that, based on W, adding sememes brings a BLEU improvement of 0.52, lower than that of 2.30 from F+W. Further, based on DeFT, adding sememes even brings noises and decreases BLEU by 0.35. This indicates that, compared with sememes, formation features are more feasible and effective.

## 6 Conclusion

In this paper, we propose to use formation features to enhance DG. We build a formation-informed dataset and design a model DeFT, which decomposes words into formation features and fuses features via a gating mechanism. Experimental results show that our method is both effective and robust.

#### Acknowledgments

We would like to thank all the anonymous reviewers and annotators for their helpful advice on various aspects of this work, including Fuqian Wu, Ming Liu, Yaqi Yin, Yue Wang, etc. This paper is supported by the National Natural Science Foundation of China (No. 62036001, U19A2065) and the National Social Science Foundation of China (No. 16YY137, 18ZDA295).

\begin{table}
\begin{tabular}{|l|l|l|l|} \hline  & **Word (w.)** & **\# \#\(]_{i}\)** & **\% \#\(]_{j}\)** \\ \cline{2-4} \multirow{5}{*}{**Event**} & **Morphh\({}_{1}\)** & **\%\(]_{1}\)**; piecemeal & **\%\(]_{1}\)**; piecemeal \\ \cline{2-4}  & **Morphh\({}_{2}\)** & **\#\(]_{1}\)**; use & **\#\(]_{2}\)**; expense \\ \cline{2-4}  & **Rule** & Adverb-Verb & Modifier-Head \\ \cline{2-4}  & **Context** & [\%\(]\)\(]\)\(\subseteq\) [w.]\(]_{0}\) 50 \(\overline{\tau}_{k}\) & **\% \(\bigotimes\) [w.]\(]_{2}\)** \\  & & (You can [w.], 50 yuan.) & (Getor [w.]\(]_{2}\)) \\ \cline{2-4}  & **Definition** & **\% \# 



## References

* Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In _ICLR 2015_.
* Bojanowski et al. (2017) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. _Transactions of the Association for Computational Linguistics_, 5:135-146.
* Cho et al. (2014) Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In _EMNLP 2014_, pages 1724-1734.
* Dong and Dong (2006) Zhendong Dong and Qiang Dong. 2006. _Hownnet and the computation of meaning_. World Scientific.
* Fleiss and Cohen (1973) Joseph L Fleiss and Jacob Cohen. 1973. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. _Educational and Psychological Measurement_, 33(3):613-619.
* Fu (1988) Yonghe Fu. 1988. Modern chinese general character list: Appendix of the most and secondly frequently used characters. _Yuwen Jianshe_.
* Gadetsky et al. (2018) Artyom Gadetsky, Ilya Yakubovskiy, and Dmitry P. Vetrov. 2018. Conditional generators of words definitions. In _ACL 2018_, pages 266-271.
* Graves and Schmidhuber (2005) Alex Graves and Jurgen Schmidhuber. 2005. Framewise phoneme classification with bidirectional LSTM and other neural network architectures. _Neural Networks_, 18(5-6):602-610.
* Ishiwatari et al. (2019) Shonosuke Ishiwatari, Hiroaki Hayashi, Naoki Yoshinaga, Graham Neubig, Shoetsu Sato, Masashi Toyoda, and Masaru Kitsuregawa. 2019. Learning to describe unknown phrases with local and global contexts. In _NAACL-HLT 2019_, pages 3467-3476.
* Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In _ICLR 2015_.
* Li et al. (2020) Jiajun Li, Yu Bao, Shujian Huang, Xinyu Dai, and Jiajun Chen. 2020. Explicit semantic decomposition for definition generation. In _ACL 2020_, pages 708-717.
* Li et al. (2018) Shen Li, Zhe Zhao, Renfen Hu, Wensi Li, Tao Liu, and Xiaoyong Du. 2018. Analogical reasoning on chinese morphological and semantic relations. In _ACL 2018_, pages 138-143.
* Lin (2004) Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In _ACL 2004 Workshop on Text Summarization Branches Out_, pages 74-81.
* Lin and Liu (2019) Zi Lin and Yang Liu. 2019. Implantating rational knowledge into distributed representation at morpheme level. In _AAAI 2019_, pages 2954-2961.
* Liu et al. (2018) Yang Liu, Zi Lin, and Sichen Kang. 2018. Towards a description of chinese morpheme conceptions and semantic composition of word. _Journal of Chinese Information Processing_, 32(2):12-21.
* Noraset et al. (2017) Thanapon Noraset, Chen Liang, Larry Birmbaum, and Doug Downey. 2017. Definition modeling: Learning to define word embeddings in natural language. In _AAAI 2017_, pages 3259-3266.
* Park et al. (2018) Hyun-jung Park, Min-chae Song, and Kyung-Shik Shin. 2018. Sentiment analysis of korean reviews using cnn: Focusing on morpheme embedding. _Journal of Intelligence and Information Systems_, 24(2):59-83.
* Washio et al. (2019) Koki Washio, Satoshi Sekine, and Tsuneaki Kato. 2019. Bridging the defined and the defining: Exploiting implicit lexical semantic relations in definition modeling. In _EMNLP-IJCNLP 2019_, pages 3519-3525.
* Wierzbicka (1996) Anna Wierzbicka. 1996. _Semantics: Primes and universals: Primes and universals_. Oxford University Press, UK.
* Yang et al. (2020) Liner Yang, Cunliang Kong, Yun Chen, Yang Liu, Qinan Fan, and Erhong Yang. 2020. Incorporating sememes into chinese definition modeling. _IEEE/ACM Transactions on Audio, Speech and Language Processing_, 28:1669-1677.
* Zhu (1982) Dexi Zhu. 1982. _Vifa Jiangyi (lectures on Grammar)_. The Commercial Press, China.
* Zhu et al. (2019) Yi Zhu, Ivan Vulic, and Anna Korhonen. 2019. A systematic study of leveraging subword information for learning word representations. In _NAACL-HLT 2019_, pages 912-932.

## Appendix A Formation-Informed Dataset Details

We show the complete descriptions of 16 formation rules in Table 7.

## Appendix B Additional Analysis on Formation Rules

Here we provide some additional interesting analyses that reveal the specific properties and influences of word formation rules.

### The similarity among different formation rules

In Section 4.2.1, we produce a comprehensive morpheme embedding using a rule-specific linear layer. We study the relations of the weight matrices \({\bf W}^{(rule)}_{m}\) in these layers by resizing them into vectors and calculating their pairwise cosine similarity, as shown in Figure 4.

Figure 4 (a) shows that Overlapping is most similar to Suffixation, Prefixation, and Single Morpheme. Interestingly, word meanings constructed

by these four formation rules share a similar pattern of using only one morpheme. For example, in Suffixation, only the first morphem carries the meaning, like "\(\overleftarrow{\text{\#}}\)" (_trick-\(\emptyset\)_).

Figure 4 (b) shows that Noun-Quantifier is most similar to Quantifier-Quantifier and Number-Quantifier. Word meanings constructed by these three formation rules all have a quantifier morpheme. For example, in Number-Quantifier, the first morpheme is a number, and the second morpheme is a quantifier, like "\(\overleftarrow{\text{\#}}\)" (_one-dot_).

Figure 4 (c) shows that Verb-Object is most similar to Modifier-Head, Parallel and Adverb. Word meanings constructed by Verb-Object, Modifier-Head and Adverb-Verb share a similar pattern of using the first morpheme to operate on the second morpheme. For example, in Adverb-Verb, the first morpheme modifies the second verb morpheme, like "\(\overleftarrow{\text{\#}}\)" (_vainly-spend_).

### The specific impact of formation rules

In Table 8, we generate definitions using different formation rules for the same word. Results show that each predicted definition indicates a clear pattern of the used formation rule. The Modifier-Head rule uses _touched_ to modify _sad_, and outputs a noun; the Adverb-Verb rule outputs an adjective in a similar modifying way; the Parallel rule outputs a single meaning of _sad_ with differ

\begin{table}
\begin{tabular}{l|l|l|r} \hline \hline
* Chinese phonetic notation
- (English translation)”. To help understand these rules, we give a simple explanation to describe the relation between two morphemes in the second column.
 

ent parts-of-speech. However, only the correct rule (Verb-Consequence) captures the **cause-and-effect** semantic aspect and outputs the correct definition. This reveals the impact of formation rules in the word formation process.

Figure 4: We take Verb-Object, Overlapping and Noun-Quantifier weight matrices as three examples and display their similarity with all the other 15 formation rules in the heatmap. The color goes deeper for more similar formation rules. The top 3 most similar formation rules are shown in **Bold**.