# A Deep Learning Approach for Sentiment Analysis of COVID-19 Reviews
## 4 Methodology
### Additional Steps in LSTM-RNN  
As previously indicated, the purpose of this project is to explore a deep learning architecture involving attention layers to understand sentiment classification accuracy from tweet data. Thus, for the proposed LSTM-RNN architecture, there are some additional processing steps. First is the convolution of features. In this phase, the input COVID-19 tweets are fed into the LSTM-RNN model. This stage aims to extract high-level semantic features from the sequence of words. The LSTM-RNN model also finds the temporal relationship between features and generates feature vectors. Additionally, we consider the semantic meaning of the tweets and create a secondary set of labels with values assigned to the emotion expressed in those tweets to train the dataset: fear = 0, sad = 1, anger = 2, and joy = 3. Acknowledging that this assignment of emotion labels is subjective, the purpose of doing so is to create a matrix layer that can be fed to LSTM-RNN towards achieving sentiment analysis outcomes. Figure 2 shows that the RNN model takes the sequence of pixels \(x=x_{1},x_{2},\ldots,x_{n}\), produces hidden states \(H=H_{1},H_{2},\ldots\ldots,H_{n}\), and outputs states \(O=O_{1},O_{2},\ldots\ldots,O_{n}\) in the following way [41; 42]:  
\[O_{t}=\sigma(W_{H_{t}O_{t}}+b_{t}) \tag{2}\]  
\[H_{t}=\sigma\big{(}W_{H_{t-1}H_{t}}H_{t-1}+W_{x_{t}H_{t}}x_{t}+b_{H_{t}}\big{)} \tag{3}\]  
Here, \(W_{H_{t}O_{t}}\) represents the vector from the hidden unit \(H_{t}\) and output unit \(O_{t}\), \(H_{t-1}\) represents the hidden unit for a \(t-1\) pixel sequence, \(W_{H_{t-1}H_{t}}\) is a weight vector from the hidden unit \(H_{t-1}\) to \(H_{t}\) for the sequence time \(t\), and \(b_{H_{t}}\) and \(b_{t}\) are biases. Figure 2 above presents a graphical model for RNN and proposed changes to indicate the improved part of the LSTM-RNN. Furthermore, the LSTM stack can be used to learn the time sequence features in which the model learns the problems comprised of a single series of observation and a model is required to learn from the series of past observations to predict the next value in sequence [12; 35]:  
\[ig_{t}=\tan\hslash(W_{x_{t}ig_{t}}x_{t}+W_{H_{t-1}ig_{t}}H_{t-1}+b_{ig_{t}}) \tag{4}\]  
\[p_{t}=\sigma\big{(}W_{x_{t}p_{t}}x_{t}+W_{H_{t-1}p_{t}}H_{t-1}+b_{p_{t}}\big{)} \tag{5}\]  
\[fg_{t}=\sigma\Big{(}W_{x_{t}fg_{t}}x_{t}+W_{H_{t-1}/g_{t}}H_{t-1}+b_{fg_{t}} \Big{)} \tag{6}\]  
\[op_{t}=\sigma\big{(}W_{x_{t}op_{t}}x_{t}+W_{H_{t-1}op_{t}}H_{t-1}+b_{op_{t}} \big{)} \tag{7}\]  
\[Ce_{t}=ce_{t-1}\odot fg_{t}+ig_{t}\odot p_{t} \tag{8}\]  
\[H_{t}=tanh(Ce_{t})\odot op_{t} \tag{9}\]  
Figure 2: A graphical model of RNN and proposed changes. where \(ig_{t}\) represents the input gate, \(p_{t}\) indicates the prediction in starting layers, \(fg_{t}\) represents the forget gate, \(H_{t}\) provides information on output, \(b_{ig},b_{p},b_{fg}\), \(b_{op}\) are bias vectors, \(Ce_{t}\) indicates the state of the cell, and \(W_{xx}\) is the weight matrix. Both RNN and LSTM models can be combined to extract semantic features from the input tweets. Notably, in the experimentation, we apply attention layer for improving the learning from features and improving the feature weights, as also noted in [39]. LSTM-RNN is used for learning sequence of sentences and generating features weighted by attention process. Further the use of secondary labels combined with LSTM-RNN facilitate the increase of domain knowledge in the learning process. As indicated in the literature [41], the focus function of the attention layer tests weight distribution and estimates an array for the different layers. Thus from \(X_{i}\) as input, f(\(X_{i}\), \(X_{i+1}\)) are the features generated from the second layer and f(\(X_{i}\), \(X_{i+1}\), \(\cdot\cdot\cdot\), \(X_{i}+L-1\)) from the \(L\)-th layer. These feature values indicate the responses of multi-scale n-grams [23; 41], i.e., unigram \(X_{i}\), bigram \(X_{i}\)\(X_{i+1}\). and L-gram \(X_{i}\)\(X_{i+1}\). \(\cdot\cdot\cdot\)\(X_{i}+L-1\).