SlowGuess/ACL_2020_OCR huggingface dataset:
Jiahuan Li * Yu Bao * Shujian Huang† Xinyu Dai Jiajun Chen
{lijh,baoy}@smail.nju.edu.cn, {huangsj,daixinyu,chenjj}  $@$  nju.edu.cn
Definition generation, which aims to automatically generate dictionary definitions for words, has recently been proposed to assist the construction of dictionaries and help people understand unfamiliar texts. However, previous works hardly consider explicitly modeling the "components" of definitions, leading to under-specific generation results. In this paper, we propose ESD, namely Explicit Semantic Decomposition for definition generation, which explicitly decomposes meaning of words into semantic components, and models them with discrete latent variables for definition generation. Experimental results show that ESD achieves substantial improvements on WordNet and Oxford benchmarks over strong previous baselines.
Dictionary definition, which provides explanatory sentences for word senses, plays an important role in natural language understanding for human. It is a common practice for human to consult a dictionary when encountering unfamiliar words (Fraser, 1999). However, it is often the case that we cannot find satisfying definitions for words that are rarely used or newly created. To assist dictionary compilation and help human readers understand unfamiliar texts, generating definitions automatically is of practical significance.
Noraset et al. (2017) first propose definition modeling, which is the task of generating the dictionary definition for a given word with its embedding. Gadetsky et al. (2018) extend the work by incorporating word sense disambiguation to generate context-aware word definitions. Both methods adopt a variant of encoder-decoder architecture,
Word	captain
Reference	the person in charge of a ship
Generated	the person who is a member of a ship
Table 1: An example of the definitions of word "captain". Reference is from Oxford dictionary and Generated is from the method of Ishiwatari et al. (2019).
where the word to be defined is mapped to a low-dimensional semantic vector by an encoder, and the decoder is responsible for generating the definition given the semantic vector.
Although the existing encoder-decoder architecture (Gadetsky et al., 2018; Ishiwatari et al., 2019; Washio et al., 2019) yields reasonable generation results, it relies heavily on the decoder to extract thorough semantic components of the word, leading to under-specific definition generation results, i.e. missing some semantic components. As illustrated in Table 1, to generate a precise definition of the word "captain", one needs to know that "captain" refers to a person, "captain" is related to ship, and "captain" manages or is in charge of the ship, where person, ship, manage are three semantic components of word "captain". However, due to the lack of explicitly modeling of these semantic components, the model misses the semantic component "manage" for the word "captain".
Linguists and lexicographers define a word by decomposing its meaning into its semantic components and expressing them in natural language sentences (Wierzbicka, 1996). Inspired by this, Yang et al. (2019) incorporate sememes (Bloomfield, 1949; Dong and Dong, 2003), i.e. minimum units of semantic meanings of human languages, in the task of generating definition in Chinese. However, it is just as, if not more, time-consuming and expensive to label the components of words than to write definitions manually.
In this paper, we propose to explicitly decom
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 708-717 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
pose the meaning of words into semantic components for definition generation. We introduce a group of discrete latent variables to model the underlying semantic components. Extending the established training technique for discrete latent variable used in representation learning (Roy et al., 2018) and machine translation tasks (van den Oord et al., 2017; Kaiser et al., 2018; Shu et al., 2019), we further propose two auxiliary losses to ensure that the introduced latent variables capture the word semantics. Experimental results show that our method achieves significant improvements over previous methods on two definition generation datasets. We also show that our model indeed learns meaningful and informative latent codes, and generates more precise and specific definitions.
In this section, we introduce the background of the original definition modeling task and two extensive works to original definition modeling.
2.1 Definition Modeling
Definition generation was firstly proposed by Noraset et al. (2017). The goal of the original task is to generate a natural language description  $\mathcal{D} = d_{1:T}$  for a given word  $w_*$ . The authors view it as a conditional language modeling task:
$$
p \left(\mathcal {D} \mid w _ {*}\right) = \prod_ {t = 1} ^ {T} p \left(d _ {t} \mid d _ {i <   t}, w _ {*}\right) \tag {1}
$$
The main drawback of Noraset et al. (2017) is that they cannot handle words with multiple different meanings such as "spring" and "bank", whose meanings can only be disambiguated using their contexts.
2.2 Word Context for Definition Modeling
To tackle the polysemous problem in the definition generation task, Gadetsky et al. (2018) introduce the task of Context-aware Definition Generation (CDG), in which a usage example  $\mathcal{C} = c_{1:|\mathcal{C}|}$  of the target word is given to help disambiguate the meaning of the word.
For example, given the word "bank" and its context "a bank account", the goal of the task is to generate a definition like "an organization that provides financial services". However, if the input context has been changed to "He jumped into the
river and swam to the opposite bank", then the appropriate definition would be "the side of a river".
They extend Eqn. 1 to make use of the given context as follows:
$$
p (\mathcal {D} | w _ {*}, \mathcal {C}) = \prod_ {t = 1} ^ {T} p \left(d _ {t} \mid d _ {i <   t}, w _ {*}, \mathcal {C}\right) \tag {2}
$$
2.3 Decomposed Semantic for Definition Modeling
Linguists consider the process of defining a word is to decompose its meaning into constituent components and describe them in natural language sentences (Goddard and Wierzbicka, 1994; Wierzbicka, 1996). Previously, Yang et al. (2019) take sememes as one kind of such semantic components, and leverage external sememe annotations HowNet (Dong and Dong, 2003) to help definition generation. They formalize the task of definition generation given a word  $w_{*}$  and its sememes  $s$  as follows:
$$
p (\mathcal {D} | w _ {*}, \boldsymbol {s}) = \prod_ {t = 1} ^ {T} p \left(d _ {t} \mid d _ {i <   t}, w _ {*}, \boldsymbol {s}\right) \tag {3}
$$
Although it is shown their method can generate definitions more accurately, they assume that annotations of sememes are available for each word, which can be unrealistic in real-world scenarios.
In this section, we present ESD, namely Explicit Semantic Decomposition for context-aware definition generation.
3.1 Modeling Semantic Components with Discrete Latent Variables
It is linguistically motivated that to define a word is to decompose its meaning into constituent components and describe them in natural language sentences (Goddard and Wierzbicka, 1994; Wierzbicka, 1996). We assume that there exists a set of discrete latent variables  $\mathbf{z} = \mathbf{z}_{1:M}$  that model the semantic components of  $w_{*}$ , where  $M$  is the hyperparameter denoting the number of decomposed components. Then the marginal likelihood of a definition  $\mathcal{D}$  that we would like to maximize given a target word  $w_{*}$  and its context  $\mathcal{C}$  can be written as follows:
$$
p _ {\theta} (\mathcal {D} | w _ {*}, \mathcal {C}) = \sum_ {\boldsymbol {z}} p _ {\theta} (\boldsymbol {z} | w _ {*}, \mathcal {C}) p _ {\theta} (\mathcal {D} | w _ {*}, \mathcal {C}, \boldsymbol {z})
$$
Figure 1: Neural architecture of ESD, including the word encoder, context encoder, the decoder and the definition encoder for the posterior networks.
However, it is generally computationally intractable to sum over all the configurations of latent variables. In order to address this issue, we instead introduce a approximate posterior  $q_{\phi}(z|w_{*},\mathcal{C},\mathcal{D})$  and optimize the evidence lower bound (ELBO) of the log likelihood  $\log p_{\theta}(\mathcal{D}|w_{*},\mathcal{C})$  for training:
$$
\begin{array}{l} \mathcal {J} _ {\mathrm {E L B O}} = \underset {q _ {\phi} (z | w _ {*}, \mathcal {C}, \mathcal {D})} {\mathbb {E}} \left[ \log p _ {\theta} (\mathcal {D} | z, w _ {*}, \mathcal {C}) \right] \\ - K L \left(q _ {\phi} \left(\boldsymbol {z} \mid w _ {*}, \mathcal {C}, \mathcal {D}\right) \mid p _ {\theta} \left(\boldsymbol {z} \mid w _ {*}, \mathcal {C}\right)\right) \\ \leq \log p _ {\theta} (\mathcal {D} | w _ {*}, \mathcal {C}) \tag {4} \\ \end{array}
$$
At the training phase, both posterior distribution  $q_{\phi}(z|w_{*},\mathcal{C},\mathcal{D})$  and prior distribution  $p_{\theta}(z|w_{*},\mathcal{C})$  are computed and  $z$  is sampled from the posterior distribution.
At the testing phase, due to the lack of  $\mathcal{D}$ , we only compute the prior distribution  $p_{\theta}(z|w_{*},\mathcal{C})$  and obtain  $z$  by applying arg max to it.
Note that for the simplicity of notions, we denote  $q_{\phi}(z_i|w_*,\mathcal{C},\mathcal{D})$  and  $p_{\theta}(z_i|w_*,\mathcal{C})$  as  $q_{i}$  and  $p_i$  in the following sections, respectively.
As shown in Figure 1,  $ESD$  is composed of three modules: the encoder stack, a decoder, and a semantic components predictor. Before detailing each component of  $ESD$ , we overview the architecture for a brief understanding.
Following the common practice of context-aware definition models (Gadetsky et al., 2018; Ishiwatari et al., 2019), we first encode the source word  $w_{*}$
into its representation  $r_*$  and context  $C = c_{1:|C|}$  into its contextual representation  $H = h_{1:|C|}$ . The semantic component predictor is responsible for predicting the semantic components  $z = z_{1:M}$ . Finally, the decoder generates the target definition from the semantic components  $z$ , the word representation  $r_*$  and the context representation  $H$ .
3.2.1 Encoder
Same as Ishiwatari et al. (2019), our encoder consists of two parts, namely word encoder and context encoder.
Word Encoder The word encoder is responsible for mapping the word  $w_{*}$  to a low-dimensional vector  $r_{*}$ , and consists of a word embedding and a character level encoder. The word embedding is initialized by large-scale pretrained word embeddings such as GloVe (Pennington et al., 2014) or FastText (Bojanowski et al., 2017), and is kept fixed at the training time. Previous works (Noraset et al., 2017; Ishiwatari et al., 2019) also show that morphological information can be helpful for definition generation. We employ a convolutional neural network (Krizhevsky et al., 2012) to encode the character sequence of the word. We concatenate the word embedding and the character encoding to get the word representation  $r_{*}$ .
Context Encoder We adopt a standard bidirectional LSTM network (Sundermeyer et al., 2012) to encode the context, which takes word embedding sequence of the context  $\mathcal{C} = c_{1:|\mathcal{C}|}$  and outputs a hidden state sequence  $H = h_{1:|\mathcal{C}|}$ .
3.2.2 Semantic Components Predictor
For the proposed ESD, we need to model both the semantic components posterior  $q_{\phi}(\pmb {z}|w_{*},\mathcal{C},\mathcal{D})$  and the prior  $p_{\theta}(z|w_{*},\mathcal{C})$
Semantic Components Posterior Approximator  
Exactly modeling the true posterior  $q_{\phi}(z|w_{*},\mathcal{C},\mathcal{D})$  is usually intractable. Therefore, we adopt an approximation method to simplify the posterior inference (Zhang et al., 2016) Following the spirit of VAE (Bowman et al., 2016), we use neural networks for better approximation in this paper.
Specifically, we first compute the representation  $H_{\mathcal{D}} = h_{1:T}^{\prime}$  of the definition  $\mathcal{D} = d_{1:T}$  with a bidirectional LSTM network. We then obtain the representation of definition  $\mathcal{D}$  and context  $\mathcal{C}$  with
max-pooling operation.
$$
h _ {\mathcal {D}} = \operatorname {m a x - p o o l i n g} \left(h _ {1: T} ^ {\prime}\right) \tag {5}
$$
$$
h _ {\mathcal {C}} = \operatorname {m a x - p o o l i n g} \left(h _ {1: | \mathcal {C} |}\right) \tag {6}
$$
With these representations, as well the word representation  $r_*$ , we compute the posterior approximation  $q_i$  of  $\mathbf{z}_i$  as follows:
$$
q _ {i} = \operatorname {s o f t m a x} \left(W _ {i} ^ {q} \left[ r _ {*}, h _ {\mathcal {C}}, h _ {\mathcal {D}} \right] + b _ {i} ^ {q}\right)
$$
where the  $W_{i}^{q}$  and  $b_{i}^{q}$  are the parameters of the semantic components posterior approximator.
Semantic Components Prior Model Similar to the posterior, we model the prior  $p_i$  of  $z_i$  by a neural network with the representation  $h_{\mathcal{C}}$  (computed by Eqn 6) and  $r_*$  as follows:
$$
p _ {i} = \operatorname {s o f t m a x} \left(W _ {i} ^ {p} \left[ r _ {*}, h _ {\mathcal {C}} \right] + b _ {i} ^ {p}\right)
$$
where the  $W_{i}^{p}$  and  $b_{i}^{p}$  are the parameters of the semantic components prior.
3.2.3 Definition Decoder
Given the word  $w_{*}$ , the context  $\mathcal{C}$  and the semantic component latent variables  $z$ , our decoder adopt a LSTM to model the probability of generating definition  $\mathcal{D}$  given word  $w_{*}$ , context  $\mathcal{C}$ , and semantic components  $z$ :
$$
p \left(\mathcal {D} \mid w _ {*}, \mathcal {C}, z\right) = \prod_ {t = 1} ^ {T} p \left(d _ {t} \mid d _ {<   t}, w _ {*}, \mathcal {C}, z\right) \tag {7}
$$
At each decoding time step, we first obtain the context vector  $c_{t}$  as follows:
$$
\alpha_ {t i} = \frac {\exp (s _ {t} ^ {T} h _ {i})}{\sum_ {j = 1} ^ {| \mathcal {C} |} \exp (s _ {t} ^ {T} h _ {j})}
$$
$$
c _ {t} = \sum_ {i} ^ {| \mathcal {C} |} \alpha_ {t i} h _ {i}
$$
Moreover, it is intuitive that at different time steps the decoder is describing different semantic perspectives of the word, thus needing different semantic components (Yang et al., 2019). We embed each  $z_{i}$  using a latent embedding matrix  $\mathrm{E}_i\in \mathbb{R}^{K\times D}$  and get  $M$  semantic component vectors  $\{E_1(z_1),E_2(z_2),\dots ,E_M(z_M)\}$ . We then apply an attention mechanism over the semantic component vectors and obtain a semantic context vector  $o_{t}$ :
$$
\begin{array}{l} \beta_ {t i} = \frac {\exp (s _ {t} ^ {T} E _ {i} (z _ {i}))}{\sum_ {j = 1} ^ {M} \exp (s _ {t} ^ {T} E _ {i} (z _ {i}))} \\ o _ {t} = \sum_ {i} ^ {M} \beta_ {t i} E _ {i} (z _ {i}) \\ \end{array}
$$
Finally, we adopt a GRU-like (Cho et al., 2014) gate mechanism to allow the decoder to dynamically fuse information from the word representation  $\mathbf{r}_{*}$ , context vector  $\mathbf{c}_t$ , and semantic context vector  $\mathbf{o}_t$ , which can be calculated as follows:
$$
\begin{array}{l} \boldsymbol {f} _ {t} = \left[ \boldsymbol {r} _ {*}; \boldsymbol {c} _ {t}; \boldsymbol {o} _ {t} \right] \\ \boldsymbol {u} _ {t} = \sigma \left(\mathbf {W} _ {u} [ \boldsymbol {f} _ {t}; \boldsymbol {s} _ {t} ] + \boldsymbol {b} _ {u}\right) \\ \boldsymbol {v} _ {t} = \sigma \left(\mathbf {W} _ {r} [ \boldsymbol {f} _ {t}; \boldsymbol {s} _ {t} ] + \boldsymbol {b} _ {r}\right) \\ \hat {\boldsymbol {s}} _ {t} = \tanh  (\mathbf {W} _ {s} [ (\boldsymbol {v} _ {t} \odot \boldsymbol {f} _ {t} ]; \boldsymbol {s} _ {t} ] + \boldsymbol {b} _ {s}) \\ \boldsymbol {s} _ {t} ^ {\prime} = \left(1 - \boldsymbol {u} _ {t}\right) \odot \boldsymbol {s} _ {t} + \boldsymbol {u} _ {t} \odot \hat {\boldsymbol {s}} _ {t} \\ \end{array}
$$
where,  $W_{*}$  and  $\pmb{b}_{*}$  are weight matrices and bias terms, respectively.
The loss function in Eqn. 4 serves as our primary training objective. Besides, since the latent variables are designed to model the semantic components, we propose two auxiliary losses to ensure that these latent variables can learn informative codes and capture the decomposed semantics.
Semantic Completeness Objective In order to generate accurate definitions, the introduces latent variables must capture all perspectives of the word semantics. For example, it is impossible to precisely define the word "captain" in the context "The captain gave the order to abandon the ship" without knowing that (1) a captain is a person, (2) a captain works in a ship, and (3) a captain usually is in charge of a ship. Therefore, an ideal  $z$  should contain sufficient information for predicting the definition.
We first propose to leverage sememe annotations of HowNet (Dong and Dong, 2003) as an external signal to guide the learning of latent variables. As we mentioned in Section 2.3, sememes are also known to be helpful for definition generation (Yang et al., 2019). Previously, Xie et al. (2017) show that it is possible to predict sememes of words from large scale pretrained distributional representations.
Suppose the set of sememes in HowNet are denoted by  $S = \{s_1, s_2, \dots, s_n\}$ , and each word  $w$  in HowNet is annotated by a small subset of  $S$ , denoted by  $S_w = \{s_i | s_i \in S\}$ . Inspired by Weng et al. (2017), we adopt a bag-of-word loss to ensure that  $z$  is informative enough to be predictive about sememe annotations  $S_w$ :
$$
\mathcal {L} _ {\mathrm {c o m}} ^ {(\mathrm {s e m})} = - \log \sum_ {s _ {i} \in S _ {w}} p \left(s _ {i} \mid z\right) \tag {8}
$$
Our next motivation is that the sememes annotation is still expensive, while definitions of words are off-the-shelf when training. Inspired by Bao et al. (2019) and John et al. (2019), we enforce the model to predict every word in the target definition  $\mathcal{D} = d_{1:T}$  to ensure that  $z$  is informative enough:
$$
\mathcal {L} _ {\mathrm {c o m}} ^ {(\mathrm {d e f})} = - \log \sum_ {i = 1} ^ {T} p \left(d _ {i} \mid z\right) \tag {9}
$$
Semantic Diversity Objective To achieve the goal of decomposing semantics, it is crucial that there are several different latent variables that separately model different semantic components. In order to prevent that multiple latent variables degenerate to one, we encourage the semantic vectors to be dissimilar from each other by introducing a disagreement loss:
$$
\mathcal {L} _ {\mathrm {d i v}} = - \sum_ {1 \leq i <   j \leq M} \operatorname {d i s t} \left(E _ {i} \left(z _ {i}\right), E _ {j} \left(z _ {j}\right)\right) \tag {10}
$$
where,  $dist(\cdot ,\cdot)$  is a distance function between two distributions. We adopt cosine distance as the distance function in this paper.
Overall Objectives With the different overall training loss used, there are two variants of ESD. The original loss of ESD is
$$
\mathcal {L} _ {\mathrm {b a s e}} = - \mathcal {J} _ {\mathrm {E L B O}}
$$
The first variant of ESD (denoted by ESD-def) includes the optimization of semantic completeness and semantic diversity, which is optimized with:
$$
\mathcal {L} _ {E S D - \mathrm {d e f}} = \mathcal {L} _ {\mathrm {b a s e}} + \alpha \mathcal {L} _ {\mathrm {c o m}} ^ {(\mathrm {d e f})} + \beta \mathcal {L} _ {\mathrm {d i v}}
$$
Grounding on the annotated sememes, the second variant of ESD (denoted by ESD-sem) is optimized with:
$$
\mathcal {L} _ {E S D - \text {s e m}} = \mathcal {L} _ {\text {b a s e}} + \alpha \mathcal {L} _ {\text {c o m}} ^ {(\text {s e m})} + \beta \mathcal {L} _ {\text {d i v}}
$$
Datasets To demonstrate the effectiveness of our method, we conduct experiments on two datasets used in previous work (Ishiwatari et al., 2019): WordNet  ${}^{1}$  and Oxford  ${}^{2}$  . Each entry in the datasets is a triple of a word,a piece of its usage example, and its corresponding dictionary definition.
Sememe Annotation Resources Following previous work (Yang et al., 2019), we take HowNet as the sememe annotation resource, which is an ontology that contains annotations for over 100,000 words with sememes. Each word in HowNet may have several senses, and each sense is annotated with several sememes explaining the meaning of it.
Hyperparameters We adopt a two-layer LSTM network as our context encoder and definition decoder. We set the hidden dim to 300. Following Ishiwatari et al. (2019), we set the CNN kernel for character encoder of length 2, 3, 4, 5, 6 and size 10, 30, 40, 40, 40 respectively with a stride of 1. The dimension of the final character level encoding is 160. We set the number of latent variables M and the number of categories K to 8 and 256, respectively.
Optimization We adopt Adam (Kingma and Ba, 2014) to optimize our model. The learning rate is set to 0.001. The  $\alpha$  and  $\beta$  we used in the overall objective are set to 1.0 and 0.1, respectively. All hyperparameters are chosen based on the performance on the validation set and are used across all the experiments.
Competitors We compare our model with several baseline models:
1. I-Attention (Gadetsky et al., 2018) uses the context to disambiguate the word embedding and cannot utilize context information at the decoding time.
2. LOG-CaD (Ishiwatari et al., 2019) is similar to our architecture, without modeling the semantic component.
3. Pip-sem is our intuitive pipeline that consists of a sememe predictor and a definition generator. The sememe predictor is trained on HowNet and is responsible for annotating words in definition generation datasets. The definition generator is used to generate definitions given the word, context, and pseudo annotations of sememes.
Metrics We adopt two several automatic metrics that are often used in generation tasks: BLEU (Papineni et al., 2002) and Meteor (Denkowski and Lavie, 2014). BLEU considers the exact match between generation results and references and is the most common metric used to evaluate generation systems. Following previous work, we compute
<sup>1</sup>https://wordnet.princeton.edu/
$^{2}$ https://en.oxforddictionaries.com/
Model	WordNet	Oxford
BLEU	METEOR	BLEU	METEOR
I-Attention (Gadetsky et al., 2018)	23.77	/	17.25	/
LOG-CaD (Ishiwatari et al., 2019)	24.79	/	18.53	/
*LOG-CaD	24.70	8.66	18.24	8.43
†Pip-sem	25.52	11.33	19.89	11.10
ESD-def	25.75	11.52	19.98	10.79
†ESD-sem	26.48	12.45	20.86	11.86
Table 2: BLEU and Meteor scores on WordNet and Oxford dataset. ‘†’ indicates models that incorporate external sememe annotations while training. ‘*’ denotes our reimplementation of previous model.
Model	Fluency	Semantic Completeness
LOG-CaD	3.53	3.01
ESD-def	3.55	3.45
Table 3: Human annotated scores on Oxford dataset.
the sentence level BLEU score. We also consider Meteor (Denkowski and Lavie, 2014), a metric that takes synonyms, stemming, and paraphrases into consideration while calculating the score. Meteor score is said to favor word choices than word orders and favor recall over precision (Denkowski and Lavie, 2014). We use the recommended hyperparameters to compute Meteor scores.
4.2 Automatic Evaluation
The results, as measured by the automatic evaluation metrics, i.e. BLEU and Meteor, are presented in Table2.
ESD significantly improves the quality of definition generation with a large margin. On all the benchmark datasets, our ESD that incorporates sememes achieves the best generation performance, both in BLEU and Meteor scores. It is worth noting that the improvement of the Meteor score is more significant than the BLEU score, i.e. 3.79 vs. 1.78 on WordNet, and 3.43 vs. 2.62 on Oxford, indicating that our model is better at recalling semantically correct words, which is consistent with our motivation to address the under-specific problem.
Decomposing semantics is indeed helpful to definition modeling. The models that generate definition with the explicit decomposed semantics (Pipsem, ESD-def and ESD-sem) leads to remarkable improvements over the competitor without decomposed component modeling (I-Attention and LOG-CaD). The comparison between the ESD-def, I-Attention and LOG-CaD is fair because all of them