# Word Sense Disambiguation Using Cosine Similarity Collaborates with Word2 vec and WordNet
## 2 Related Works
### Word2 vec  
Word2 vec (2018) is a popular word-embedding approach and has many advantages compared to the earlier algorithms reported by [PERSON] et al. (2018). Word2 vec represents a word with a fixed-size vector. This vector is capable of effectively capturing semantic and syntactic similarities. For example, v(\"queen\") - v(\"woman\") + v(\"man\") \(\approx\) v(\"king\") or v(\"Thailand\") + v(\"capital\") \(\approx\) v(\"Bangkok\"). Word2 vec is a prediction-based approach that trains from a large text corpus. Word2 vec is one hidden layer artificial neural network. There are two models: Continuous bag-of-words (CBOW) and skip-gram. CBOW is given a context, then predicts a target word. However, skip-gram is just the mirror of CBOW; it is given a target word then predicts a context. In this paper, we will briefly describe the skip-gram only. The input of skip-gram is a single target word \(w_{I}\), and the output is \(w_{I}\)'s context \([w_{O,I},w_{O,Z},\ldots,w_{O,C}]\) defined by window size \(C\). For example, we take the sentence, \"I go to the bank to deposit money.\" If a target word is \"bank\", window size 4, stopword \"to\" and \"the\" are removed, the context will be \"I go... deposit money.\" The training instances, the input word, and the output words for the skip-gram model are formed with the following tuples: (\"bank\", \"I\"), (\"bank\", \"go\"), (\"bank\", \"deposit\"), and (\"bank\", \"money\"). Each word is encoded to one-hot vector size \(V\). \(V\) is the size of the vocabulary. All positions will be \"0\", except the position corresponding with the word which will be \"1\". For example, in the vocabulary list (\"I\", \"go\", \"bank\", \"deposit\", \"money\"), the one-hot vector for \"bank\" is (0,0,1,0,0). These one-hot vectors are viewed as a column matrix with a size \(V\times 1\).  
From Figure 1, \(x\) is a one-hot vector corresponding to the input word, and \(y\) is a probability vector corresponding to an output word in the training corpus. \(V\times N\) matrix \(W\) is the weight matrix between the input layer and the hidden layer. Each row is a word-embedding vector size \(N\) that we will use to retrieve a word vector later. \(N\) is the training parameter of the model (i.e., dimensions of the word-embedding vectors). A hidden layer \(h\) is a multiplication matrix between one-hot vector \(x\) and matrix \(W\) with a size \(1\times N\), that makes the hidden layer \(h\) equivalent to matrix \(W\) row \(i\) which is mathematically expressed as Equation (1).  
\[h=x^{T}W=W_{(i,)} \tag{1}\]  
\(N\times V\) matrix \(W^{\prime}\) is a weight matrix between the hidden layer and the output layer. The output layer is a multiplication matrix between hidden layer \(h\) and weight matrix \(W^{\prime}\) with a size \(1\times V\), each component of the output layer is computed by Softmax classifier which produces the probability distribution represented by Equation (2).  
\[\begin{split} u_{j}&=hW^{\prime}_{(.,)}\\ P\Big{(}w_{O,j}\big{|}w_{I}\Big{)}&=y_{j}=\frac{ \exp\big{(}u_{j}\big{)}}{\sum_{j-1}^{V}\exp\big{(}u_{j}\big{)}}\end{split} \tag{2}\]  
Figure 1 illustrated only one training instance; actually, there are a bunch of training instances that can input to the model. The objective is to learn the weight \(W\) and \(W^{\prime}\) that minimizes the loss function (i.e., the difference between system outputs and output instances) by using backpropagation and the stochastic gradient descent algorithm [21]. There are many techniques involved in the training processes, such as subsampling frequent words and negative sampling [10; 22; 23]. After training, the word-embedding matrix \(W\) was used to retrieve a word vector.