# Roman Urdu Sentiment Analysis Using Transfer Learning
## 3 Methodology
### CNN Stocked with Attention Layer and Transfer Learning
#### 3.1.3 Attention Mechanism  
All the words in a sentence do not contribute the same amount to the meaning of the sentence, and because the pooling layer in a CNN causes local features to be lost, there needs to be a way to draw attention to words that have a bigger effect on the meaning of a sentence when context and word interactions are taken into account.  
Extracting feature maps from the same-sized filter is used to conduct the attention mechanism on the convolutional layer. Let us pretend that M distinct region widths are considered, and m distinct filters are used in the convolution layer. Therefore, a \(H_{ij}eR^{h_{i}\times d}\) filter is applied to the sentence matrix K, where \(i\) = 1, 2,..., \(M\), and j = 1, 2,..., \(m\), \(M\), to produce an m-by m feature map. This new sentence matrix \(X_{i}eR^{n\times m}\) (Equation (3)) is obtained by connecting feature maps from filters with the same size. Each cell in this matrix represents a feature that can be extracted from the input using filters of the same size. n represents the number of words.  
\[X_{i}=\begin{bmatrix}x_{1,1}&S&x_{1,m}\\ M&O&M\\ x_{n-F_{i}+1,1}&K&x_{n-F_{i}+1,m}\end{bmatrix}, \tag{3}\]  
It is the function of the attention mechanism to assign each row a relative importance for the sake of retrieving the relevant information from the text. To accomplish this, we first feed the new word matrix \(X_{i}\) into a single layer perceptron with the parameters \(WeR^{m\times d}\) and \(U_{i}eR^{n-h_{i}+1\times d}\). This yields a hidden representation of the matrix, which we then use to predict new words (Equation (4)).  
\[U_{i}=\tanh(X_{i}W+b), \tag{4}\]  
To find the normalized importance weight \(a_{i}eR^{n-h_{i}+1\times 1}\), we compare each word's importance to its similarity to the context vector \(ueR^{d\times 1}\) below (Equation (5)). As a process similar to that utilized by memory networks [23; 33], the context vector \(u\) can be seen as a symbolic depiction of the informative words.  
\[a_{i}=softmax(U_{i}u), \tag{5}\]  
In particular, u starts at 0 and is learned throughout training so that each row in \(\overline{X}_{ii}\) Xi is assigned the same weight. Then, we obtain a different representation of \(X_{i}\) by multiplying each element of \(a_{i}\) by the row in the \(X_{i}\) matrix that it corresponds to (is the element-wise product) (Equation (6)).  
\[\overline{X}_{i}=a_{i}oX_{i}, \tag{6}\]  
when the attention mechanism is applied to a representation of \(X_{i}\), a new representation of \(X_{i}\) is created, with the informative words precisely identified. The \(X_{i}\) matrix is created by blending feature maps with identical filter sizes.  
Then, a single-layer perceptron is used to generate a new representation of \(X_{i}\) (called \(U_{i}\)). In training, a hyper-parameter is adjusted to maximize the correlation between \(U_{i}\) and the content vector \(u\), which is used to determine the normalized importance weight \(a_{i}\) that indicates the significance of each word. Multiplying each element of \(a_{i}\) by the row number in \(X_{i}\) yields a new representation of \(\overline{X}_{i}\). Using the attention mechanism typically results in the extraction of more informative words.