## 3 Background
### Skip-Gram  
The skip-gram model assumes that a word can be used to predict the words that come before and after it in a text sequence. For example, consider the text sequence \"we\", \"rise\", \"by\", \"lifting\", \"others\". As shown in Figure 3, if we choose \"by\" as the center word and set the size of the context window to 2, then the SG model considers the conditional probability of generating the context words from the left side of the center word \"we\" and \"rise\" and, from the right side, \"lifting\" and \"others\"; \(P(\"we\", \"rise\", \"lifting\", \"others\" |\"by\")\), The \"window size\" parameter is used to limit the number of words in each context (see Table 3). Suppose that we represent the input of those example words; the output of the model will be a vector containing the probability that a randomly selected nearby word is each word in our vocabulary. Figure 2: Architecture of the Word2 vec training model. (**a**) CBOW and (**b**) SG model architectures. In the SG model training context, for example, if input target word \(f(i)=\) \"by\" is given to the model, there will be four target context words, i.e., our window size is \((\text{max}\textit{context location }C=2)\) (Context window size \(\text{K }=4\)), with four activation functions, four predictions, and four errors summed to obtain the total error. The network will tell us the probability that every word in our vocabulary \(V\) is the nearby word we choose for the window size. The vocabulary here is built from the sentence. For example, if we have 5 unique words and use \"by\" as a hot encoding vector, the vector will have 5 components (one for each word in our vocabulary) and place \"1\" in a position corresponding to the word \"by\" which is a unique spot and \"0\" in all other positions. The dot product in the hidden layer (no activation function at this point) passes to the output layer. The output layer computes the dot product between the hidden layer's output vectors and weighs the output layer matrix. Then, the softmax function computes the probability of a word appearing in the context of \(f(i)\) at a given context location, which helps us distribute the probability throughout the output nodes, as we want to convert out the last layer output in terms of probability. More formally, the SG model learns the representation of words by maximizing the objective function over the entire set of \(S\)_sentences_, defined as follows. \[\mathcal{L}=\sum_{s\in S}\sum_{f_{i}\in s}-\sum_{c\,\,j\,\leq\,c,j\
eq 0}log\ p \big{(}f_{i+j}|f_{i}\big{)}, \tag{1}\]  
where \(c\) is the training context window size (that may depend on \(f_{i}\)) and words from the same sentences are ordered arbitrarily. Probability \(p\big{(}f_{i+j}|f_{i}\big{)}\) of observing a neighboring word \(f_{i+j}\) given the current word \(f_{i}\) is defined using the well-known softmax function,  
\[p\big{(}f_{i+j}|f_{i}\big{)}=\frac{exp\big{(}u_{f_{i}}^{T}v_{f_{i+j}}\big{)}}{ \sum_{f=1}^{F}exp\big{(}u_{f_{i}}^{T}v_{f}\big{)}}, \tag{2}\]  
where \(u_{f}\) and \(v_{f}\) are the vector representations of word \(f\) as input and output, respectively, \(F\) is the number of unique words in the entire vocabulary. According to Equations (1) and (2), the SG model captures the context of the word sequence, so that words with similar contexts will have similar vector representations. The numerator in Equation (2) demonstrates this by giving a larger value for similar words through the dot product of the two vectors. If the words do not occur in each other's context, the representations will be different, and thus the numerator will be a small value. Figure 3: The SG model considers the probability of predicting the surrounding context words given a center word. \begin{table}
\begin{tabular}{c|c|c|c|c} \hline \hline
**Window Size** & \multicolumn{2}{c}{**Text**} & \multicolumn{2}{c}{**Training Samples**} \\ \hline  & & & & (by, we) \\
2 & we & rise & by & lifting & others \\ \hline \hline \end{tabular}
\end{table}