###### Abstract  
Obtaining high-quality embeddings of out-of-vocabularies (OOVs) and low-frequency words is a challenge in natural language processing (NLP). To efficiently estimate the embeddings of OOVs and low-frequency words, we propose a new method that uses the dictionary to estimate the embeddings of OOVs and low-frequency words. More specifically, the explanatory note of an entry in dictionaries accurately describes the semantics of the corresponding word. Naturally, we adopt the sentence representation model to extract the semantics of the explanatory note and regard the semantics as the embedding of the corresponding word. We design a new sentence representation model to encode sentences to extract the semantics from the explanatory notes of entries more efficiently. Based on the assumption that the higher quality of word embeddings will lead to better performance, we design an extrinsic experiment to evaluate the quality of low-frequency words' embeddings. The experimental results show that the embeddings of low-frequency words estimated by our proposed method have higher quality. In addition, both intrinsic and extrinsic experiments show that our proposed sentence representation model can represent the semantics of sentences well.  
natural language processing; word embedding; BERT; dictionary 20211108 22 October 2021 19 November 2021 21 November 2021 2021 22 October 2021 21 November 2021 22 October 2021 21 November 2021  
to out-of-vocabulary (OOV) everywhere. For some tasks that require word segmentation, the OOV phenomenon is more obvious. This is because word segmentation leads to more significant long-tail characteristics [15; 16]. In addition, [PERSON]'s law applies to most languages. This makes word embedding models unable to fully learn the semantics of OOVs and low-frequency words [16; 17]. Therefore, accurately estimating the embeddings of OOVs and low-frequency words becomes the research motivation of this paper. We take Chinese as an example and use a dictionary to estimate the embeddings of those words. Different from English texts, there are no explicit delimiters such as whitespace to separate words in Chinese texts [15; 18], just like the explanatory note in Figure 1. Therefore, Chinese word segmentation is important for some Chinese NLP tasks [15; 18]. However, Chinese word segmentation will cause more serious sparsity problems, which makes the embeddings of OOVs and low-frequency words more difficult to be estimated [16].  
An entry in the dictionary contains a word and the corresponding explanatory note. They all point to the same point in the semantics space. As shown in Figure 1, the explanatory note usually contains rich information which explains the meaning of the corresponding word exactly. Inspired by this, we designed a semantics extractor to extract semantics from explanatory notes. We use the semantics representation produced by the extractor as the representation of low-frequency words. For high-frequency words, we still retain their word representations estimated by other word embedding models, such as Word2 Vec. By combining the two types of word embedding estimation methods, we will obtain higher quality word representations that will be fine-tuned in downstream tasks. As the extrinsic experimental results in this paper show, the higher the quality of the word representation, the better the performance we will obtain. Our main contributions are as follows:  
* We use the dictionary to estimate the embeddings of OOVs and low-frequency words. We also study the effects of low-frequency word embedding replacement rate on the performance of semantics match tasks.
* We propose a new sentence representation model which is different from the current mainstream LM, such as BERT [1], XLNet [19], and GPT [2; 20].