## 3 Methodology
### Word Embedding  
#### 3.3.1 GloVe  
Word embedding aims to transform textual information into a vector of real value. Semantic/language vector space models of language represent each word with a real-valued vector. Several techniques have been proposed to represent words in real-valued vectors, such as TF-IDF [10; 45], Latent Semantics Analysis (LSA) [43], and neural network models such as word2 vec [13] and GloVe [15].  
Word vectorization is divided into global matrix factorizations such as LSA [46] and local context windows like the Skip-Gram model [13]. According to [10; 43], both approaches have drawbacks. Global matrix factorization effectively utilized the statistical information, but it fails to capture word analogy. On the other hand, the Window-based model successfully captures word analogy but poorly uses global statistics.  
The bag of words model is exploited by TF-IDF concepts [10; 43]. TF-IDF relies on the statistical information of words from several documents. Terms refer to a word or a group of words; TF stands for term frequency. TF is basically the frequency of words in a document; to normalize the value, the frequency is divided by the number of words in a single document. IDF stands for inverse document frequency. The value of IDF is a logarithmic value of the number of documents divided by the TF.  
GloVe [15] explores the global representation of the entire corpus and incorporates the meaning of the words in this. Word frequency and co-occurrence are the main metrics by which the values of real-valued vectors of particular words are calculated. GloVe is an unsupervised method, where there is no human to introduce ground-truth meaning into the collection of words (corpus). The basis of calculation is the exploitation of the frequency of particular words and the closest words surrounding each word.  
In GloVe, the first step is collecting the most frequent words as the context. The second step is to scan the words in the corpus to build a co-occurrence matrix \(X\). Let us consider \(i\) as the index of frequent words and \(j\) as the rest of the words in the corpus. \(P_{ij}\) is the probability of a word \(j\) occurring with the context word \(i\).  
\[P_{ij}=P(j|i)=\frac{X_{ij}}{X_{i}} \tag{1}\]  
Let us consider two words \(i\) and \(j\), as well as a context word \(k\); we can calculate a ratio of co-occurrence probability as follows:  
\[F\big{(}w_{i},w_{j}\widetilde{w}_{k}\big{)}=\frac{P_{ij}}{P_{jk}} \tag{2}\]  
Finally, the loss function \(J\) can be calculated as follows:  
\[J=\sum_{i,j=1}^{V}\!\!f\big{(}X_{ij}\big{)}\left(W_{i}^{T}\ \widetilde{w}_{j}+b_{i}+b_{j}-log\ X_{ij}\right)^{2}, \tag{3}\]  
\begin{table}
\begin{tabular}{c c} \hline
**Word Embedding** & **LSTM Architectures** \\ \hline \multirow{3}{*}{GloVe} & Single-Layer LSTM \\ \cline{2-3}  & Double-Layer LSTM \\ \cline{2-3}  & Triple-Layer LSTM \\ \hline \multirow{3}{*}{fastText} & Single-Layer LSTM \\ \cline{2-3}  & Double-Layer LSTM \\ \cline{1-1} \cline{2-3}  & Triple-Layer LSTM \\ \hline \end{tabular}
\end{table}
Table 2: Research scenarios.  
where \(f\) is the weighting function. The training aims to minimize the least squares error. Once GloVe is trained, every word is assigned to a particular real-valued vector.