# Fusion Text Representations to Enhance Contextual Meaning in Sentiment Classification
## 3 Proposed Work
### GloWord_biGRU
#### 3.2.1 Embedding Layer
The objective of skip-gram is to maximize the average log probability:  
\[J_{s}=\frac{1}{T}\sum_{t=1}^{T}\sum_{-c\leq j\leq c,j\
eq 0}\log p\big{(}w_{t+ j}|w_{t}\big{)} \tag{3}\]  
where \(J_{s}\) is the objective function to be maximized, \(T\) is the total number of words in the corpus, and \(c\) is the window size, representing the number of context words on each side of the target word \(w_{t}\). \(\log p\big{(}w_{t+j}|w_{t}\big{)}\) is the log probability of the context word \(w_{t+j}\) given the target word \(w_{t}\). In the CBOW model, the algorithm predicts the target word based on the context word within the window. The objective is to maximize the probability of the target word given the surrounding context words:  
\[J_{c}=\frac{1}{T}\sum_{t=1}^{T}\log p(w_{t}|w_{t-c},\ldots,\;w_{t-1},\;w_{t+1}, \ldots,\;w_{t+c}) \tag{4}\]  
In both architectures, the probabilities \(p(w_{O}|w_{t})\) are defined using the softmax function:  
\[p(w_{O}|w_{t})=\frac{\exp\left(v_{w_{O}}^{T}v_{w_{t}}\right)}{\sum_{w=1}^{V}exp (v_{w}^{T}v_{w_{t}})} \tag{5}\]  
where \(v_{w_{t}}\) and \(v_{w_{O}}\) are the input and output vector representations of words. Through this training process, Word2 Vec learns word vectors that capture semantic and syntactic relationship based on the context in which words appears. By combining \(E_{GloVe}\) and \(E_{Word2 Vec}\) into the embedding matrix \(E\), we can harness the complementary strengths of both embedding techniques, leading to richer and more informative word representations for downstream neural network model. In our proposed model, let \(V\) be the vocabulary size and \(d\) be the embedding dimension; then, the embedding matrix for GloVe can be represented as \(E_{GloVe}\in\mathbb{R}^{V\times d}\). Here, each word \(w_{i}\) is represented by the embedding vector \(E_{GloVe}(w_{i})\in\mathbb{R}^{d}\). Meanwhile, Word2 Vec is a neural network-based embedding method that learns word representations by maximizing the probability of neighboring words in a given context. The formula for the embedding matrix in Word2 Vec can be represented \(E_{Word2 Vec}\in\mathbb{R}^{V\times d}\). Here, each word \(w_{i}\) is represented by the embedding vector \(E_{word2 Vec}(w_{i})\in\mathbb{R}^{d}\). Once all the embedding matrices are ready, the next step is to build a fusion model to combine these two embedding matrices. Suppose we have input text \(x\) that has been processed into a sequence of words \(\{w_{1},w_{2},\ldots,w_{T}\}\), where \(T\) is the maximum length of the sequence. The model for each embedding can be seen below:  
\[x_{GloVe}= \begin{bmatrix}E_{GloVe}(w_{1})\\ E_{GloVe}(w_{2})\\ \ldots\\ E_{GloVe}(w_{T})\end{bmatrix} \]  
with \(x_{GloVe}\in\mathbb{R}^{T\times d}\)  
\[x_{Word2 Vec}= \begin{bmatrix}E_{word2 Vec}(w_{1})\\ E_{word2 Vec}(w_{2})\\ \ldots\\ E_{word2 Vec}(w_{T})\end{bmatrix} \]  
with \(x_{Word2 Vec}\in\mathbb{R}^{T\times d}\). Once all the embedding matrices are ready, the next step is to build an embedding fusion, which is a combination of \(x_{GloVe}\) and \(x_{Word2 Vec}\). \[x=Concatenate(x_{GloVe},x_{Word2 Vec}) \tag{6}\]  
with \(x\in\mathbb{R}^{T\times 2d}\), where concatenation is performed in the embedding dimension so that each word is represented by a vector embedding of dimension \(2d\). Mathematically, the embedding layer transforms the sequence of words \(\{w_{1},w_{2},\ldots,w_{T}\}\) into a sequence of embedding vectors \(\{e_{1},e_{2},\ldots,e_{T}\}\):  
\[e_{i}=[E_{GloVe}(w_{i})\parallel E_{Word2 Vec}(w_{i})] \tag{7}\]  
where \(e_{i}\), is the concatenated embedding for the word \(w_{i}\), and \(\parallel\) denotes the concatenation operator.