# Sentiment Analysis Based Requirement Evolution Prediction
## 3 Joint Text Analysis Model
### LSTM-Based Sentiment Analysis  
In this subsection, we use LSTM-based sentiment analysis technology to identify the user's opinions being expressed towards each product feature. LSTM is a powerful type of recurrent neural network, which is to capture intra-sentence relations to provide additional valuable information for sentiment prediction. In our problem, we intend to use pre-trained word embedding in the LSTM-based RNN framework for the sentence-level sentiment classification task, which forms the basis for the next step to identify hierarchical features with sentiment orientation.  
Figure 1: The joint model framework (LSTM+HLDA).  
In the prevailing architecture of LSTM networks, there are an input layer, a recurrent LSTM layer, and an output layer. In the input layer, each word from sentences is mapped to a \(D\)-dimensional vector, which is initialized by pre-trained word embedding vectors. Given an input sequence of sentence \(s=(features,sentiments)\), compositional context vector \(x_{t}\) is created by mapping each word token in \(s\) to a \(D\)-dimensional vector. This is designed to capture context dependencies between the features and sentiment of the sentence. The input layer can be represented as:  
\[X=[h_{t-1},x_{t}] \tag{1}\]  
where the \(h_{t}\) is the vector of the hidden layer.  
The LSTM layer connects to the input layer. The high-level compositional representations are learned by passing through the concatenated word embedding vector of the input layer to LSTM layers. LSTM incorporates \"gates\", which are neurons that use an activation function to remember certain input adaptively and are multiplied with the output of other neurons. It is capable of capturing relations among words and remembering important information for sentiment prediction. The LSTM-based recurrent architecture we used in this work is described as follows:  
\[f_{t} =\sigma(W_{f}\cdot X+b_{f}) \tag{2}\] \[i_{t} =\sigma(W_{i}\cdot X+b_{i})\] (3) \[o_{t} =\sigma(W_{o}\cdot X+b_{o})\] (4) \[c_{t} =f_{t}\odot c_{t-1}+i_{t}\odot tanh(W_{c}\cdot X+b_{c}) \tag{5}\]  
where \(f_{t},i_{t},o_{t},c_{t}\) are the forget gate, input gate, output gate, and memory cell, respectively. \(W\) is the weighted matrices. \(b\) is the bias of LSTM to be learned during training. \(\sigma\) is the sigmoid function. \(\odot\) is element-wise multiplication.  
The cell output units are connected to the output layer of the LSTM network for sentiment classification using softmax. We put the last hidden vector \(h_{t}\) of the representation of a sentence in a softmax layer.  
\[h_{t}=o_{t}\odot tanh(c_{t}) \tag{6}\]  
The probability of softmax sentiment classification in the output:  
\[p(y_{t}=k|s,\theta)=\frac{exp(W_{k}^{T}h_{t})}{\sum_{k=1}^{K}exp(W_{k}^{T}h_{t})} \tag{7}\]  
where \(w_{k}\) are the weights between the last hidden layer and the output layer. For sentence-level sentiment analysis, we use the Negative Log Likelihood (NLL) to fit the models from the training data by minimizing its value for the sentence sequence.  
\[L(\theta)=\sum_{t=1}^{T}\sum_{k=1}^{K}y_{tk}\log p(y_{t}=k|s,\theta) \tag{8}\]  
where \(y_{tk}=I(y_{t}=k)\) is an indicator variable. The loss function minimizes the cross-entropy between the predicted distribution and the target distribution.  
Based on such an LSTM recurrent architecture, we can capture sentiment information in the review sentence and identify polarities of sentiments.