# A New Sentiment-Enhanced Word Embedding Method for Sentiment Analysis
## 2 Related Work  
### Word Embedding  
Word embedding is a knowledge enhancement technique because it can capture syntactic and semantic information in textual corpora. Word2 Vec [1] has two models: a continuous skip-gram model (Skip-gram) and a continuous bag-of-words model (CBOW). They all use sliding windows to capture contextual information between words. Word2 Vec achieved the best results on the Semantic-Syntactic Word Relationship dataset and the Microsoft Sentence Completion Challenge dataset. However, Word2 Vec could not get global information of words. GloVe [2] used a co-occurrence probability matrix to obtain the global information of words. It also used slide windows to capture contextual information between words. Experimental results showed that GloVe achieved state-of-the-art performance on word analogies, similarity, and named entity recognition tasks. Since word embeddings trained by Word2 Vec and GloVe could only assign a vector to each word, they could not solve the ambiguity problem of words. In addition, Word2 Vec and GloVe could only capture the contextual information of words in a window due to the limited computing power. With the increased computing power, the embedding from language models (ELMo) [29] used two reversed recurrent neural network (RNN) layers to encode one input sequence. For a word \(i\), the forward RNN captured the above information of the word \(i\), and the backward RNN captured the following information of the word \(i\). By concatenating the forward hidden state and backward hidden state of the word \(i\), the contextual information of the word \(i\) was obtained. The ELMo model achieved remarkable performance on CoNLL 2003 NER and CoNLL 2000 chunking tasks.  
Due to the improvements in computing power, BERT [3] used bidirectional transformers [30] to encode information of a whole sentence. Since the self-attention mechanism [30] can capture the information of an entire sequence, the contextual information between words is no longer limited to the sliding window. BERT broke all records for the general language understanding evaluation (GLUE), with an average improvement of 6.7% over the other models. Moreover, BERT achieved significant performance on SQuAD, CoNLL-2003, and SWAG datasets. [PERSON] et al. [31] averaged Chinese and English word embeddings in multilingual BERT (m-BERT) [3], respectively. They obtained the Chinese-English difference vector by subtracting the average English vector from the average Chinese vector. The experimental results showed that a Chinese word embedding minus the difference vector could get English word embedding and vice versa.  
Even though word embedding has shown its power in many natural language processing tasks, it cannot distinguish the sentiment orientation of semantically similar words well. [PERSON] et al. [32] extracted hashtags, mentions, and keywords from tweets. They performed centrality-aware random walks on them to get their word/node embeddings. Then, they put the embeddings into a deep learning model for sentiment classification. The performance of their method is better than those of the original baseline models. Sent2 Vec [33] used the smoothed inverse frequency (SIF) [34] or the unsupervised smoothed inverse frequency (uSIF) [35] to encode a sentence into a vector. Then, this vector is input into a multi-layer perceptron (MLP) to learn sentimental embedding. These methods could achieve good results based on the syntactic and semantic knowledge inside a dataset in tasks of different sentiment classifications. However, these methods could not utilize external knowledge about the dataset.