# Fusion Text Representations to Enhance Contextual Meaning in Sentiment Classification
## 3 Proposed Work
### GloWord_biGRU
#### 3.2.1 Embedding Layer  
GloVe and Word2 Vec are two embedding techniques that transform words into high-dimensional vectors that encapsulate the semantic relationships and contextual nuance between words. Glove leverages co-occurrence statistics from a corpus to generate word  
Figure 1: Proposed GloWord_biGRU Architecture. vectors, capturing both local and global statistical information. In contrast, word2 vec employs a shallow, two-layer neural network to produce embeddings by predicting word context (Skip-gram) or the current word based on its context (Continuous Bag of Words, CBOW). In the context of neural networks, these word embeddings serve as foundational inputs, providing a dense, continuous representation of words that facilitate better performance on various NLP tasks, especially in sentiment classification. First, we represent the embeddings for both word embeddings as \(E=[E_{GloVe},E_{Word2 Vec}]\). Here, \(E\) denotes the combination embedding matrix, where \(E_{GloVe}\) and \(E_{Word2 Vec}\) are the individual embedding vectors generated by GloVe and Word2 Vec models, respectively. This combined representation is utilized as input for subsequent neural network models to leverage the strengths of both embedding techniques. \(E_{GloVe}\) is the embedding matrix derived from GloVe model. The GloVe algorithm constructs word vectors by factoring in the co-occurrence probability of words withing a large corpus. Specifically, it builds a co-occurrence matrix \(X\), where each entry \(X_{ij}\) indicates how often word \(i\) appears in the context of word \(j\). The model then minimizes a weighted least squares objective function to learn word vectors \(w_{i}\) and context \(\widetilde{w}_{j}\) such that their dot product approximates the logarithm of the word-word co-occurrence probabilities:  
\[J=\sum_{i,j=1}^{V}f\big{(}X_{ij}\big{)}\left(w_{i}^{T}\widetilde{w}_{j}+b_{i} +\widetilde{b}_{j}-\log X_{ij}\right)^{2} \tag{1}\]  
where \(J\) is the objective function to be minimized, \(V\) is the vocabulary size (the number of words in the corpus), \(w_{i}\) is the word vector for word \(i\), \(\widetilde{w}_{j}\) is the context vector for word \(j\), and \(b_{i}\) and \(\widetilde{b}_{j}\) are the bias terms associated with the word and context, respectively. \(X_{ij}\) is the entry in the co-occurrence matrix that indicates how frequently word \(i\) appears in the context of word \(j\). \(f\big{(}X_{ij}\big{)}\) is the weighting function used to assign smaller weights to word pairs with very high or very low co-occurrence frequencies. In GloVe, \(f\big{(}X_{ij}\big{)}\) is a weighting function used to adjust the contribution of word pairs based on their co-occurrence frequency \(X_{ij}\). This function plays a crucial role in reducing the influence of very high or very low frequencies, making the model more stable in learning word representations. The function \(f\big{(}X_{ij}\big{)}\) is generally defined as follows:  
\[f\big{(}X_{ij}\big{)}=\left\{ \begin{array}{c}\big{(}\frac{X_{ij}}{X_{max}} \big{)}^{\alpha},\;\;if\;X_{ij}<X_{max}\\ 1,\;\;if\;X_{ij}\geq X_{max}\end{array} \right. \tag{2}\]  
where \(X_{max}\) is the maximum threshold value, beyond which the contribution of co-occurrence frequency does not increase. \(\alpha\) is a parameter that controls the shape of the weighting function (usually set between 0.5 and 1). This \(f\big{(}X_{ij}\big{)}\) function ensures that word pairs with very low frequencies do not contribute too little, and those with very high frequencies do not dominate the training process. Meanwhile, \(E_{Word2 Vec}\) represents the embedding matrix obtained from the Word2 vec model. Word2 vec offers two main architectures for generating word embeddings: Skip-gram and Continuous Bag of Words (CBOW). In the Skip-gram model, the algorithm predicts the context words given a target word within a fixed window size.