# A Deep Learning-Based Approach for Multi-Label Emotion Classification in Tweets
## 3 Methodology
### BNet: System Description
#### 3.2.2 Encoding Module  
The goal of the encoding module is to map the sequence of word representations \(\{v_{w_{1}},v_{w_{2}},\ldots,v_{w_{l}}\}\) that is obtained from the embedding module to a single real-valued dense vector. In this work, we used a Recurrent Neural Network (RNN) to design our encoder. RNN reads the input sequence of vectors in a forward direction (left-to-right) starting from the first symbol \(v_{w_{1}}\) to the last one \(v_{w_{l}}\). Thus, it processes sequences in temporal order, ignoring the future context. However, for many tasks on sequences, it is beneficial to have access to future, as well as to past information. For example, in text processing, decisions are usually made after the whole sentence is known. The Bidirectional Recurrent Neural Network (BiRNN) variant [13] proposed a solution for making predictions based on both past and future information.  
A BiRNN consists of forward \(\overrightarrow{\phi}\) and backward \(\overleftarrow{\phi}\) RNNs. The first one reads the input sequence in a forward direction and produces a sequence of forward hidden states \((\overrightarrow{h_{1}},\ldots,\overrightarrow{h_{l}})\), whereas the former reads the sequence in the reverse order \((v_{w_{l}},\ldots,v_{w_{1}})\), resulting in a sequence of backward hidden states \((\overleftarrow{h_{l}},\ldots,\overleftarrow{h_{1}})\).  
We obtained a representation for each word \(v_{w_{t}}\) by concatenating the corresponding forward hidden state \(\overrightarrow{h_{t}}\) and the backward one \(\overleftarrow{h_{t}}\). The following equations illustrate the main ideas:  
\[\overrightarrow{h_{t}} =\overrightarrow{\phi}(v_{w_{t}},\overrightarrow{h_{t-1}}) \tag{8}\] \[\overleftarrow{h_{t}} =\overleftarrow{\phi}(v_{w_{t}},\overleftarrow{h_{t+1}})\] (9) \[h_{t} =[\overrightarrow{h_{t}};\overleftarrow{h_{t}}] \tag{10}\]  
The final input representation of the sequence is:  
\[c=F(\{h_{1},h_{2},\ldots,h_{l}\}) \tag{11}\]  
We simply chose \(F\) to be the last hidden state (i.e., \(F(\{h_{1},h_{2},\ldots,h_{n}\})=h_{n}\)).  
In this work, we used two Gated Recurrent Units (GRUs) [14], one as \(\overleftarrow{\phi}\) and the other as \(\overleftarrow{\phi}\). This kind of RNN was designed to have more persistent memory, making them very useful to capture long-term dependencies between the elements of a sequence. Figure 3 shows a graphical depiction of a gated recurrent unit.  
A GRU has _reset_ (\(r_{t}\)) and _update_ (\(z_{t}\)) gates. The former can completely reduce the past hidden state \(h_{t-1}\) if it finds that it is irrelevant to the computation of the new state, whereas the later is responsible for determining how much of \(h_{t-1}\) should be carried forward to the next state \(h_{t}\).  
The output \(h_{t}\) of a GRU depends on the input \(x_{t}\) and the previous state \(h_{t-1}\), and it is computed as follows:  
\[r_{t}=\sigma\left(W_{r}\cdot[h_{t-1};x_{t}]+b_{r}\right) \tag{12}\] \[z_{t}=\sigma\left(W_{h}\cdot[h_{t-1};x_{t}]+b_{z}\right)\] (13) \[\widetilde{h}_{t}=tanh\left(W_{h}\cdot[(r_{t}\odot h_{t-1});x_{t }]+b_{h}\right)\] (14) \[h_{t}=(1-z_{t})\odot h_{t-1}+z_{t}\odot\widetilde{h}_{t} \tag{15}\]  
In these expressions, \(r_{t}\) and \(z_{t}\) denote the _reset_ and _update_ gates, \(\widetilde{h}_{t}\) is the candidate output state, and \(h_{t}\) is the actual output state at time \(t\). The symbol \(\odot\) stands for element-wise multiplication; \(\sigma\) is a sigmoid function; and; stands for the vector-concatenation operation. \(W_{r},W_{z},W_{h}\in\mathbb{R}^{d_{h}\times(d+d_{h})}\) and \(b_{r},b_{z},b_{h}\in\mathbb{R}^{d_{h}}\) are the parameters of the _reset_ and _update_ gates, where \(d_{h}\) is the dimension of the hidden state and \(d\) is the dimension of the input vector.