# DAWE: A Double Attention-Based Word Embedding Model with Sememe Structure Information
## 1 Introduction  
The basis of applying deep learning to solve natural language processing (NLP) tasks is to obtain high-quality representations of words from large amounts of text data [1]. Traditionally, words are represented in a sparse high-dimensional space using count-based vectors in which each word in a vocabulary is represented by a single dimension [2]. In contrast, word embedding aims to map words into continuous low-dimensional semantic space; in this way, each word is represented by a real-valued vector, namely word vector, often composed tens or hundreds of dimensions [3]. Word embedding assumes that words used in similar ways should have similar representations, thereby naturally capturing their meaning. Word vectors obtained from word embedding have been widely used in many applications: text summarization, sentiment analysis, reading comprehension, machine translation, etc. In a great deal of word embedding-related works that have emerged in recent years, the Word2 Vec [3] model strikes a good balance between efficiency and quality. In the trainingprocess of Word2 vec, words are mapped to the same vector space. For words that share a similar context in the corpus, their corresponding vectors should be close to each other in the vector space [4]. During the training process, the vector assignments of the words are repeatedly adjusted until the values are close enough to each other, if they are adjacent in the text corpus. As a result, the low-frequency words in corpus cannot be accurately represented in Word2 Vec model because the training process of such words is not sufficient. Recently, studies have shown that taking external knowledge as the complement of text corpus can effectively improve the quality of word embeddings [5; 6; 7; 8; 9; 10]. Among them, the \"word-sense-sememe\" (Figure 1) knowledge is an intuitive form of organizing words and their senses that are easily organized and understood [11]. By synthesizing the \"word-sense-sememe\" knowledge, [PERSON] et al. [7] proposed the sememe-encoded word representation learning (SE-WRL) model that made significant performance in word embedding. SE-WRL model believes that contributions of each sememe under a sense are equivalent. However, the nature of sememes determines that different sememes under a sense may be different, which means the contributions of each sememe to the sense should be varied depending on the particular case. The inequality may be caused by two main reasons: (1) Different senses correspond to the different hierarchical structures of sememes. The sememes are organized into a hierarchical structure, such as the sememes in the sense \"Apple brand\" in Figure 1. Because of the hierarchical structure, there is fusion among sememes, which means that sememes at different branches of different levels are usually not equivalent. For example, the sememe \"computer\" in the sense \"Apple brand\" can be presented by its under-layer sememes (\"PatternValue\", \"able\", \"bring\" and \"SpeBrand (specific brand)\"). Furthermore, sememes at the same level are not equivalent in most cases. (2) The context of sememes is varied. The meaning of a word needs to be reflected in a special context, so the sememes are also affected by the context of the word. As shown in Figure 2, when the word \"Apple\" appears in the context \"I am going to the \(\sim\) store now.\", the meaning of \"Apple\" should be close to the sense \"Apple brand\". At this point, the sememe \"SpeBrand\" should have a higher weight than other sememes of the sense \"Apple brand\". Therefore, the weights of sememes in a sense should change dynamically with different contexts. Figure 1: An example of the word–sense–sememe structure, where sememes are defined as minimum units of word meanings. There exists a limited close set of sememes to compose the open set of word meanings (i.e., word sense). For instance, the word (the first layer) “Apple” contains three senses (the second layer): “Apple Brand” (a famous computer brand), “Apple” (a sort of fruit) and “Apple Tree”.