# Senti-Sequence: Learning to Represent Texts for Sentiment Polarity Classification
## 3 The Proposed Classification Framework
### Bidirectional LSTM for Document Polarity Identification  
The aim of this task consists of obtaining a single representative polarity for the whole document. To accomplish this, first, the polarity distribution within a text is represented as a sequence of marks denoting the sentiment of each word. These marks result from the polarity prediction task performed by the classifier proposed in Section 3.1. Then, an ad hoc classifier capable of dealing with these sequential data is employed to obtain the overall document polarity. The complete process is depicted in the second stage of Figure 1. As a special case, when no embedding is available for a given word, its mark in the sequence is a 0.  
Without loss of generality, let us consider that there exists an embedding vector for each word present in the underlying text corpus. Then, each input document \(t\) containing a sequence of \(n_{t}\) words \([w_{1},w_{2},\ldots,w_{n_{t}}]\) is transformed into a binary vector  
Figure 2: Example of the document sequence generation procedure from its word polarities.  
\([\phi(w_{1}),\phi(w_{2}),\dots,\phi(w_{n_{t}})]\) by the trained polarity classifier, where each \(\phi(w_{i})\in\{-1,+1\}\) is defined in Section 3.1. The main idea behind our proposal is to build a dense vector representation that relates the polarity value of each input word with the polarity values of all its surrounding words within the document. In order to accomplish this task, we make use of the bidirectional recurrent network learning formulation. This is, to estimate a function \(G\) that takes \(\phi(w_{i})\) and outputs real values  
\[\overrightarrow{h_{i}} =G(\phi(w_{i}),\overrightarrow{h}_{i-1})\] \[\overleftarrow{h_{i}} =G(\phi(w_{i}),\overleftarrow{h}_{i+1})\]  
to account for the nonlinear influence of polarities of words appearing before \(w_{i}\) and after in the text, respectively. The effect of this procedure is to map each binary value \(\phi(w_{i})\) onto a continuous polarity value that also considers the word context represented by its neighborhood within the text.  
In the end, we need to estimate another function \(f\) that merges these two output values and predicts the overall text polarity.  
These networks have long been used to model sequential data in natural language processing tasks, and despite their wide usage, they present problems with unstable gradients and very limited short memory. To overcome the last issue, long short-term memory (LSTM) is employed due to its capability of selecting, preserving, and extracting important inputs whenever needed. Bidirectional architectures have also been successfully employed for similar tasks in recent works [33,42].  
Regarding some practical issues, a zero padding strategy is used to address the variable length of text documents. Additionally, each word polarity is processed by the network in order of appearance in the text and fed to the network. The underlying idea of providing contextual information for each word polarity is mainly carried by the long-term states of the two opposite LSTM layers. Figure 3 depicts the proposed architecture.  
Furthermore, we pose that the binary valued polarity of each word is enriched by adding information of the entire text. Additionally, it is important to note that we use several recurrent units; hence, each binary polarity value is mapped onto several dimensions.  
Figure 3: Bidirectional LSTM Architecture.