Sentence model based subword embeddings for a dialog system

ETRI JournalEarly View ORIGINAL ARTICLEFree Access Sentence model based subword embeddings for a dialog system Euisok Chung, Corresponding Author Euisok Chung eschung@etri.re.kr orcid.org/0000-0001-5091-2508 Integrated Intelligence Research Section, Electronics and Telecommunications Research Institute, Daejeon, Republic of Korea Correspondence Euisok Chung, Integrated Intelligence Research Section, Electronics and Telecommunications Research Institute, Daejeon, Republic of Korea. Email: eschung@etri.re.krSearch for more papers by this authorHyun Woo Kim, Hyun Woo Kim Integrated Intelligence Research Section, Electronics and Telecommunications Research Institute, Daejeon, Republic of KoreaSearch for more papers by this authorHwa Jeon Song, Hwa Jeon Song Integrated Intelligence Research Section, Electronics and Telecommunications Research Institute, Daejeon, Republic of KoreaSearch for more papers by this author Euisok Chung, Corresponding Author Euisok Chung eschung@etri.re.kr orcid.org/0000-0001-5091-2508 Integrated Intelligence Research Section, Electronics and Telecommunications Research Institute, Daejeon, Republic of Korea Correspondence Euisok Chung, Integrated Intelligence Research Section, Electronics and Telecommunications Research Institute, Daejeon, Republic of Korea. Email: eschung@etri.re.krSearch for more papers by this authorHyun Woo Kim, Hyun Woo Kim Integrated Intelligence Research Section, Electronics and Telecommunications Research Institute, Daejeon, Republic of KoreaSearch for more papers by this authorHwa Jeon Song, Hwa Jeon Song Integrated Intelligence Research Section, Electronics and Telecommunications Research Institute, Daejeon, Republic of KoreaSearch for more papers by this author First published: 12 May 2022 https://doi.org/10.4218/etrij.2020-0245AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract This study focuses on improving a word embedding model to enhance the performance of downstream tasks, such as those of dialog systems. To improve traditional word embedding models, such as skip-gram, it is critical to refine the word features and expand the context model. In this paper, we approach the word model from the perspective of subword embedding and attempt to extend the context model by integrating various sentence models. Our proposed sentence model is a subword-based skip-thought model that integrates self-attention and relative position encoding techniques. We also propose a clustering-based dialog model for downstream task verification and evaluate its relationship with the sentence-model-based subword embedding technique. The proposed subword embedding method produces better results than previous methods in evaluating word and sentence similarity. In addition, the downstream task verification, a clustering-based dialog system, demonstrates an improvement of up to 4.86% over the results of FastText in previous research. 1 INTRODUCTION Recently, large-scale transfer learning has produced promising results in deep-learning-based language processing [1-3], indicating that large learning parameters and long learning times are essential to system performance. However, further research is required to apply large-scale transfer learning to dialog systems. This is because modern dialog systems depend on multiple domains, knowledge bases, and modalities and thus require more detailed approaches. Word embedding techniques that can be easily used in various network designs are a potential solution. In particular, [4] and [5] used word embedding techniques in sentence generation to obtain impressive results. Specifically, [4] integrated word embeddings into the input/output space to capture and utilize the output structure of the language. In addition, [5] used independently learned word embeddings to train a sequence model. The word embedding technique played the role of a teacher network in the knowledge distillation training method, which is a type of neural network learning method. In [5], the performance of word embeddings had a great influence on the performance of the sequence model. In this paper, we attempt to replace the sequence model with a dialog model. We create a target based on word embeddings in the dialog model and then train and evaluate the dialog model based on this. To this end, we first focus on improving the performance of word embedding technology. We propose a new subword embedding technique that considers sentence information and utilizes a clustering-based dialog system methodology for verification. A typical word embedding approach, such as the skip-gram model [6], employs high-capacity text as learning data. However, a problem with the skip-gram technique is that it learns word embeddings only for a given lexical list. This problem is solved by a method called subword embedding, which expresses a given word as a set of specific sub-features and performs embedding to learn the sub-features [7]. Another problem is to determine how to augment word vector models for the task of sentence representation. The reason for needing to augment the word vector models is that most language processing systems require sentence context processing. In particular, for a dialog system, a word embedding technique capable of matching the input and output sequences of sentences is essential, and the clustering-based dialog system proposed in this study can be a means of verifying the word embedding technique. The proposed word embedding technique is based on subword information and integrates the sentence model by using the skip-thought model [8]. In this study, various experimental results are presented in the sentence model, and self-attention [9] and relative position encoding [10] are considered. Korean is used as the target language; therefore, the position encoding technique for Korean syntactic words (Eojeol) is also verified. The clustering-based dialog system can be used as a verification platform for various word embedding technologies. The dialog set response list is converted into sentence representations through word embedding, and response clusters are generated by K $$ K $$ -means clustering [11]. By encoding a unified sequence of the given context and question, the decoder responds with the correct sentence and simultaneously predicts the response class. This is useful for validating word and sentence embedding techniques suitable for a dialog system. In particular, in the case of dialog domains, the scope of responses is limited. Therefore, a clustering approach based on sentence embeddings for the corresponding responses is available. It is thus considered that the dialog response prediction performance is closely related to the sentence embedding performance. We constructed 7236 conversation sets 11 https://fashion-how.org/ETRI/board.html. from a Korean dialog-based clothing recommendation domain to evaluate the dialog system. The contributions of this study are as follows: This study demonstrates that the proposed subword embedding technique using sentence information from various evaluation sets leads to improved results compared with existing techniques. The evaluation set consists of the lexical level, sentence level, and dialog context level. The clustering-based dialog system indicates the importance and utilization of word embedding technology. In Section 2, we discuss related work and describe subword embedding techniques and dialog models. Finally, we provide various experimental results and conclusions. 2 RELATED WORK 2.1 Word embedding approaches Recent research related to word embedding involves the application of text mining and improvements to word embedding. Specifically, research on text mining to extract latent knowledge from large volumes of data in scientific journals has recently been conducted [12]. In addition, studies on improving word embedding have proposed changes to the context model through dependency structuring [13], multi-meaning embedding [14, 15], a meta-embedding technique for integrating various word embeddings [16], and out-of-vocabulary problem-solving [7, 17, 18]. This study starts by reviewing word embedding technology that uses subwords to solve out-of-vocabulary problems. In a previous study [19], FastText [7] exhibited more stable subword embedding performance than byte-pair embeddings [20]. In the present study, we integrate a sentence model into FastText, and the sentence model integration follows the skip-thought [8] structure. However, the sentence model is different from the skip-thought model because instead of performing sentence embedding using long short-term memory [21], the sentence model is composed of only subword parameters. Additionally, similar to the skip-gram context model, similar sentences within a specific distance are selected as the context sentence. Word embedding using our sentence model is similar to the Siamese CBOW model [22]; namely, it embeds the surrounding sentences and target sentences into the constituent word embedding average values and reflects the cosine similarity of the sentence embedding pairs to word embedding learning. However, we use the skip-gram model instead of the CBOW model for the sentence model and use subwords as the embedding unit. In terms of multi-prototype word embeddings, we do not model directly as in previous studies [14, 15] but take an approach that considers the multi-meaning at the point of application of word embedding. We achieve this approach through self-attention and relative position encoding. In this study, Korean is the target language. In subword embedding for English, the subword features can be composed of alphabetic characters. Similarly, the subword embedding for Korean can use syllables (characters), phonemes, and graphemes. According to a previous study [23], there was no significant difference between syllable and phoneme subword results. Furthermore, in [17], syllable and grapheme subword features exhibited similar results; however, by using two features in combination, an increase in the word embedding performance was reported. In this study, we use syllables as subword features because the type of subword feature is not the focus of this paper. We use the evaluation sets provided by [17] for the evaluation of word similarity and word analogy. In addition, we construct a sentence similarity evaluation set and examine the relationship between word embedding and the dialog system using the dialog evaluation set. 2.2 Dialog model approaches A dialog system can be largely classified into a task-based approach and a non-task-based approach [24]. However, even for the task-based approach, most recent research has followed an end-to-end approach [25, 26]. In the field of response generation, retrieval-based [27-29], generation-based [30], and hybrid-based [31-33] approaches have been studied. Recently, approaches to reduce the search space of a conversation interaction by clustering the components of the conversation set have been attempted [34-36]. In particular, [35] performed clustering through utterance embedding and proposed a method of tracking and predicting an utterance class. The dialog model in the present study is similar to the approach proposed in [35]. However, the difference is that we cluster only the response results and employ a predictive approach when decoding. In addition, a knowledge network related to the dialog context is added to the encoder, while for multi-task learning, a response generation function is added to the decoder. This study focuses mainly on the relationship between word embedding and dialog modeling. FIGURE 1Open in figure viewerPowerPoint (A) Subword skip-gram model (SSGM) and (B) subword skip-thought model (SSTM). Here, Φ t $$ {\Phi}_t $$ is a set of subword vectors for the SSGM target word and SSTM target sentence, while Φ c $$ {\Phi}_c $$ is a set of subword vectors for the SSGM context word. Φ n $$ {\Phi}_n $$ is for the SSTM target sentence. In addition, z f t $$ {z}_f^t $$ is the subword vector. The subscript f $$ f $$ is the subword feature of the target word. The subword features are the same in all three tables, Φ t $$ {\Phi}_t $$ , Φ c $$ {\Phi}_c $$ , and Φ n $$ {\Phi}_n $$ , with the vector values of the subword features being learned simultaneously. In the verification of the learned subword embeddings, we conduct an experiment using only Φ t $$ {\Phi}_t $$ (CV) and an experiment integrating Φ t $$ {\Phi}_t $$ , Φ c $$ {\Phi}_c $$ , and Φ n $$ {\Phi}_n $$ (IV) (see Section 5.1) 3 SUBWORD MODEL The main topic of this paper is how to project a sentence context into subword vectors. The skip-gram model [37] is a prototype of the proposed model and describes a probability model in which the target word predicts nearby words. In this study, we perform word embedding by extending the skip-gram model to a subword-based approach, referred to as the subword skip-gram model (SSGM). Then, sentence modeling is performed using a subword skip-thought model (SSTM). 3.1 Integration of the sentence model The structure of the two models is presented in Figure 1, which illustrates the connection between the two models with the shared subword parameters Φ t $$ {\Phi}_t $$ . Equation (1) describes a model that integrates SSGM and SSTM. The goal of this study is to determine the subword vectors Φ t $$ {\Phi}_t $$ that maximize the log-likelihood L $$ \mathcal{L} $$ . Here, T w $$ {\mathcal{T}}_w $$ denotes the size of the subword embedding learning data W $$ W $$ , while T s $$ {\mathcal{T}}_s $$ denotes the size of the skip-thought learning data S $$ S $$ . C t $$ {\mathcal{C}}_t $$ is a set of context words w c $$ {w}_c $$ for the target word w t $$ {w}_t $$ , while N t $$ {\mathcal{N}}_t $$ is a set of context sentences s n $$ {s}_n $$ for the target sentence s t $$ {s}_t $$ . L = ∑ t = 1 T w ∑ c ∈ C t log p w c | w t + ∑ t = 1 T s ∑ n ∈ N t log p s n | s t . $$ \mathcal{L}=\sum \limits_{t=1}^{T_w}\sum \limits_{c\in {C}_t}\log \kern.6em p\left({w}_c|{w}_t\right)+\sum \limits_{t=1}^{T_s}\sum \limits_{n\in {N}_t}\log \kern.6em p\left({s}_n|{s}_t\right). $$ (1) SSGM. Equation (2) is an exponential language model using the score function q $$ q $$ of w t $$ {w}_t $$ and w c $$ {w}_c $$ . The existing subword model [7] uses only the constituent features F t $$ {\mathcal{F}}_t $$ of the target word w t $$ {w}_t $$ in (3), and the context word w c $$ {w}_c $$ uses the word vector v c $$ {v}_c $$ . In (3), f $$ f $$ is one of the many subword features of w t $$ {w}_t $$ . F t $$ {\mathcal{F}}_t $$ is a set of the subword features of w t $$ {w}_t $$ , and z f $$ {z}_f $$ is the subword vector for f $$ f $$ . Thus, f ∈ F t $$ f\in {\mathcal{F}}_t $$ and z f ∈ Φ t $$ {z}_f\in {\Phi}_t $$ . p w c | w t = e q ( w t , w c ) ∑ i = 1 T w e q ( w t , w i ) , $$ p\left({w}_c|{w}_t\right)=\frac{e^{q\left({w}_t,{w}_c\right)}}{\sum_{i=1}^{{\mathcal{T}}_w}{e}^{q\left({w}_t,{w}_i\right)}}, $$ (2) q w t , w c = ∑ f ∈ F t z f ⊤ · v c . $$ q\left({w}_t,{w}_c\right)=\sum \limits_{f\in {\mathcal{F}}_t}{z}_f^{\top}\cdotp {v}_c. $$ (3) We modify v c $$ {v}_c $$ in (3) as the sum of subword vectors z g $$ {z}_g $$ to reduce the computational burden because the number of vocabularies is generally very large. Equation (4) describes these modifications. It makes a relation, f ∈ F c $$ f\in {\mathcal{F}}_c $$ and z g ∈ Φ c $$ {z}_g\in {\Phi}_c $$ . The score function q $$ q $$ of (4) is a dot product of two vectors, each using the sum of the subword vectors. q w t , w c = ∑ f ∈ F t z f t ⊤ · ∑ g ∈ F c z g c . $$ q\left({w}_t,{w}_c\right)={\left(\sum \limits_{f\in {\mathcal{F}}_t}{z}_f^t\right)}^{\top}\cdotp \left(\sum \limits_{g\in {\mathcal{F}}_c}{z}_g^c\right). $$ (4) SSTM. Equation (5) describes a skip-thought model in which the lexical object of a skip-gram model is converted into a sentence object. This differs from the previous approach [8] because we select the bilinear model of sentence vectors through subword embeddings rather than a recurrent neural network (RNN). Thus, unlike the case of an RNN, all learning results are stored in the target Φ t $$ {\Phi}_t $$ and context subword vectors Φ n $$ {\Phi}_n $$ . p s n | s t = e q ( s t , s n ) ∑ i = 1 T s e q ( s t , s i ) . $$ p\left({s}_n|{s}_t\right)=\frac{e^{q\left({s}_t,{s}_n\right)}}{\sum_{i=1}^{{\mathcal{T}}_s}{e}^{q\left({s}_t,{s}_i\right)}}. $$ (5) Equation (6) describes the score function of a target sentence s t $$ {s}_t $$ and context sentence s n $$ {s}_n $$ . Each sentence vector is calculated as an average of the constituent word vectors. Because the word vector is calculated as the sum of the constituent subword vectors, the learning result is reflected in Φ t $$ {\Phi}_t $$ and Φ n $$ {\Phi}_n $$ . q s t , s n = ∑ w t ∈ s t ∑ f ∈ F t z f t s t ⊤ · ∑ w n ∈ s n ∑ g ∈ F n z g n s n . $$ q\left({s}_t,{s}_n\right)={\left(\frac{\sum_{w_t\in {s}_t}{\sum}_{f\in {\mathcal{F}}_t}{z}_f^t}{\left|{s}_t\right|}\right)}^{\top}\cdotp \left(\frac{\sum_{w_n\in {s}_n}{\sum}_{g\in {\mathcal{F}}_n}{z}_g^n}{\left|{s}_n\right|}\right). $$ (6) For learning, the computational complexity of the denominators in (2) and (5) is excessive. The general solution is to convert the problem into a binary classification problem, such as Monte Carlo noise-contrastive estimation [38]. Negative samples for w t $$ {w}_t $$ and s t $$ {s}_t $$ are generated using negative models q $$ q $$ and q ′ $$ {q}^{\prime } $$ , respectively, and the equation for modeling the binary classification is described in (7). Learning is performed by maximizing this formula. L NCE k MC = ∑ ( w c , w t ) ∈ W log p ( D = 1 | w c , w t ) + ∑ i = 1 , w t — ∼ q k log p ( D = 0 | w c , w t — ) + ∑ ( s c , s t ) ∈ S log p ( D = 1 | s c , s t ) + ∑ i = 1 , s t — ∼ q ′ k log p ( D = 0 | s c , s t — ) . $$ {\mathcal{L}}_{{\mathrm{NCE}}_k}^{\mathrm{MC}}=\sum \limits_{\left({w}_c,{w}_t\right)\in W}\left(\log \kern0.5em p\left(D=1|{w}_c,{w}_t\right)+\sum \limits_{i=1,\overset{\hbox{---} }{w_t}\sim q}^k\log \kern0.5em p\Big(D=0|{w}_c,\overset{\hbox{---} }{w_t}\Big)\right)+\sum \limits_{\left({s}_c,{s}_t\right)\in S}\left(\log \kern0.5em p\left(D=1|{s}_c,{s}_t\right)+\sum \limits_{i=1,\overset{\hbox{---} }{s_t}\sim {q}^{\prime}}^k\log \kern0.5em p\Big(D=0|{s}_c,\overset{\hbox{---} }{s_t}\Big)\right). $$ (7) 3.2 Extension of the sentence model In the SSTM, the sentence model uses the mean of the word vector values and combines a variety of approaches. We extend the model according to self-attention [9], relative position encoding [10], and Korean sentence structure. Self-attention. We utilize the query–key–value ( Q K V $$ \mathcal{QKV} $$ ) self-attention described in [9]. First, the sentence is converted into a matrix S $$ \mathcal{S} $$ of word vectors, and the matrix is set to a constant value for each Q $$ \mathcal{Q} $$ , K $$ \mathcal{K} $$ , and V $$ \mathcal{V} $$ . Here, the attention operation, softmax ( QK ⊤ / d k ) V $$ \mathrm{softmax}\left({\mathcal{QK}}^{\top }/\sqrt{d_k}\kern0.2em \right)\mathcal{V} $$ , is performed, and the matrix V $$ \mathcal{V} $$ is updated using the result. In this process, we do not use additional learning parameters, such as linear layers for Q $$ \mathcal{Q} $$ , K $$ \mathcal{K} $$ , and V $$ \mathcal{V} $$ . Finally, the average of S $$ \mathcal{S} $$ and the self-attention result V $$ \mathcal{V} $$ is used. Relative position encoding. The authors of [10] proposed two relative position encoding parameters, rpe v $$ {\mathrm{rpe}}^v $$ and rpe k $$ {\mathrm{rpe}}^k $$ , which are integrated with the self-attention operations. In the softmax operation, rpe v $$ {\mathrm{rpe}}^v $$ is added to V $$ \mathcal{V} $$ according to the relative distance of words, and r p e k $$ rp{e}^k $$ is added to K $$ \mathcal{K} $$ . Both values consist of n $$ n $$ vectors according to the relative distance n $$ n $$ , a hyperparameter. Let an element α i j $$ {\alpha}_{ij} $$ of softmax ( QK ⊤ / d k ) $$ \mathrm{softmax}\left({\mathcal{QK}}^{\top }/\sqrt{d_k}\kern.2em \right) $$ be the attention weight of the j th $$ j\mathrm{th} $$ word vector with respect to the i th $$ i\mathrm{th} $$ word vector of the sentence matrix V $$ \mathcal{V} $$ . In this case, rpe d ( i , j ) v $$ {\mathrm{rpe}}_{d\left(i,j\right)}^v $$ is added to V j $$ {\mathcal{V}}_j $$ according to the relative distance d ( i , j ) $$ d\left(i,j\right) $$ between V i $$ {\mathcal{V}}_i $$ and V j $$ {\mathcal{V}}_j $$ , and the attention operation is performed. Here, rpe d ( i , j ) k $$ {\mathrm{rpe}}_{d\left(i,j\right)}^k $$ is a value added to K j $$ {\mathcal{K}}_j $$ according to the relative distance d ( i , j ) $$ d\left(i,j\right) $$ during the Q K ⊤ $$ \mathcal{Q}{\mathcal{K}}^{\top } $$ operation. Sentence model for Korean. Because Korean is an agglutinative language, grammatical morphemes are integrated into lexical morphemes to form syntactic words called Eojeol. 22 Two Eojeols are provided as an example. "" can be read as "dari-ro ga" and translated to English as "go to the bridge." Here "(dari-ro)" means "to the bridge" and "(ga)" means "go." Furthermore, "(dari)" means "bridge," and "-(-ro)" is a postpositional particle. The Korean sentence model adds an Eojeol layer, which is an intermediate step between a word and sentence that assigns weights according to positions of the words constituting the Eojeol. An Eojeol is described as e $$ e $$ in (8), where the weight of the word is expressed by η $$ \eta $$ . Because the number of words in each Eojeol is different, the first lexical morpheme receives the highest value, while the remaining morphemes receive the same value. q s t , s n = ∑ e t ∈ s t ∑ w t ∈ e t η w t s t · ∑ f ∈ F t z f t s t e ⊤ · ∑ e n ∈ s n ∑ w n ∈ e n η w n s n · ∑ g ∈ F n z g n s n e . $$ {\displaystyle \begin{array}{cc}\hfill q\left({s}_t,{s}_n\right)& ={\left(\frac{\sum_{e_t\in {s}_t}{\sum}_{w_t\in {e}_t}\left[{\eta}_{w_t}^{s_t}\cdotp {\sum}_{f\in {F}_t}{z}_f^t\right]}{{\left|{s}_t\right|}^e}\right)}^{\top}\cdotp \hfill \\ {}\hfill & \kern2em \left(\frac{\sum_{e_n\in {s}_n}{\sum}_{w_n\in {e}_n}\left[{\eta}_{w_n}^{s_n}\cdotp {\sum}_{g\in {F}_n}{z}_g^n\right]}{{\left|{s}_n\right|}^e}\right).\hfill \end{array}} $$ (8) 4 CLUSTERING-BASED DIALOG MODEL The purpose of the dialog model is to evaluate the subword embedding approach. From another point of view, we examine the relationship between utterance clustering-based dialog models and word embedding techniques. A sentence is described as a vector with the mean value of the constituent word embeddings. K $$ K $$ -means clustering is performed using the sentence vector. Here, similar sentences in vector space are located close to each other, while different sentences are located far from each other. The vector space becomes the clustering representation space. One drawback of the model proposed in this paper is that it does not consider the validity of clustering. However, because clustering is applied to learning and evaluation in a limited domain, and response sentences of a limited type are targeted for clustering, it is assumed that the clustering error is insignificant. The proposed dialog approach is illustrated in Figure 2. FIGURE 2Open in figure viewerPowerPoint Clustering-based dialog model. The history, knowledge, and response are represented as subword embeddings. FC, fully connected First, the dialog dataset is described. In the i th $$ i\mathrm{th} $$ step of a conversation, the history is H i = { s i − n , s i − n + 1 , . . . , s i } $$ {\mathcal{H}}_i=\left\{{s}_{i-n},{s}_{i-n+1},\dots, {s}_i\right\} $$ , where s i $$ {s}_i $$ is the utterance at time i $$ i $$ . Knowledge consists of m $$ m $$ sentences with K i = { k 1 , k 2 , . . . , k m } $$ {\mathcal{K}}_i=\left\{{k}_1,{k}_2,\dots, {k}_m\right\} $$ . The response A i $$ {\mathcal{A}}_i $$ is s i + 1 $$ {s}_{i+1} $$ . The history encoder is a bidirectional multilayer gated recurrent unit (GRU) [39] that converts the unified sequence of H i $$ {\mathcal{H}}_i $$ into subword form and receives it at the input layer. The knowledge encoder receives as input K i $$ {\mathcal{K}}_i $$ consisting of m $$ m $$ sentences determined by H i $$ {\mathcal{H}}_i $$ at time i $$ i $$ . Here, K i $$ {\mathcal{K}}_i $$ is converted into m ′ ( < m ) $$ {m}^{\prime}\left(<m\right) $$ sentences through a filtering process by measuring the similarity with the sequence embedding of H i $$ {\mathcal{H}}_i $$ . They are integrated, scaled, and combined with the final state of the history encoder and passed through the fully connected (FC) layer to the decoders. The response decoder consists of a multilayer GRU. Multiple attentions are applied to the history encoder and knowledge encoder in different decoder layers. The response decoder then performs response generation. For the response class prediction block, the response list of the dialog dataset is converted into the response class by K $$ K $$ -means clustering in advance. The response class prediction infers response class C ( A i ) $$ \mathcal{C}\left({\mathcal{A}}_i\right) $$ using the state information transmitted from the encoders. Training the dialog model attempts to maximize (9). L = ∑ i log p A i | H i , K i + log p C A i | H i , K i . $$ \mathcal{L}=\sum \limits_i\log \kern.5em p\left({\mathcal{A}}_i|{\mathcal{H}}_i,{\mathcal{K}}_i\right)+\log \kern.5em p\left(\mathcal{C}\left({\mathcal{A}}_i\right)|{\mathcal{H}}_i,{\mathcal{K}}_i\right). $$ (9) 5 EXPERIMENTS 5.1 Evaluation of subword embedding This section describes our evaluation of the sentence-model-based subword embedding performance. We used the word similarity, word analogy, and sentence similarity evaluation sets. 5.1.1 Settings Evaluation data. The word similarity evaluation set was the released version of the WS353 evaluation set (WS) provided in [40] translated into Korean [17]. 33 https://github.com/SungjoonPark/KoreanWordVectors. It used the average similarity of two words (353 pairs) assigned on a scale of 0 to 10 by 10 or more evaluators. Word embedding values for the word pairs were expressed as two vectors, where the performance was evaluated by comparing the cosine similarity of the two vectors with the values assigned by the human graders. The authors of [17] also released the word analogy evaluation set (WA), which consisted of 5000 items for evaluating semantic features and 5000 items for evaluating syntactic features. Additionally, the WA set with 10 000 items was the Korean translation of the English word analogy test sets [37]; however, [17] applied a considerable degree of modification to the evaluation set reflecting the characteristics of the Korean language. We constructed an evaluation set for our sentence similarity (SS) experiments. This set was composed of 1000 sentence pairs consisting of 7 to 15 words each. The average on five scales of the values assigned by 10 evaluators was used as the similarity of the sentence pairs. In this study, we compared the similarity assigned to the sentence evaluation set and the cosine similarity of the two sentence embedding vectors. TABLE 1. The words and sentences in the training corpus word_seg are composed of morphemes as a result of word segmentation and are used for the subword skip-gram model Dataset Words (#) Sentences (#) Wikipedia 55.8 M 4.6 M Online news 46.2 M 3.5 M Online interview 27.2 M 2.2 M Total 125.8 M 10.3 M Total w o r d _ s e g $$ word\_\mathit{\operatorname{seg}} $$ 213.7 M 10.3 M Total w o r d _ s e g + s e n t _ d i v $$ word\_\mathit{\operatorname