# Using BiLSTM Networks for Context-Aware Deep Sensitivity Labelling on Conversational Data
## 3 Background and Approach
### LSTM Networks  
First, we define the BiLSTM Recurrent Neural Network (RNN) more formally. A recurrent neural network is a special type of normal artificial neural network (ANN) which is capable of modelling sequential data by having recurrent connections [25]. In essence, it maintains a hidden state, which can be considered as a \"memory\" of previous inputs. This is driven by the fact that each neuron represents an approximation function of all previous data.  
Figure 1 illustrates the architecture of a simple RNN. The input units \(\{\ldots,x_{t-1},x_{t},x_{t+1},\ldots\}\) where \(x=(x_{1},x_{2},x_{3},\ldots,x_{N})\), are connected to the hidden units \(h_{t}=(h_{1},h_{2},\ldots,h_{M})\) in the hidden layer, via connections defined by weight matrix \(W_{IH}\). Every hidden unit is connected to the next one with recurrent connections given by \(W_{HH}\). Each hidden unit is therefore formulated by:  
\[h_{t}=f_{H}(o_{t}) \tag{1}\]  
where:  
\[o_{t}=W_{IH}+W_{HH}h_{t-1}+b_{h} \tag{2}\]  
\(F_{h}\) is a non-linear function such as tanh, ReLU or sigmoid, etc., and \(b_{H}\) is the bias vector. The hidden layer is also connected with the output layer with weights \(W_{HO}\). Lastly the outputs \(y_{t}=(y_{1},y_{2},\ldots,y_{P})\) are defined by:  
\[y_{t}=f_{O}(W_{HO}h_{t}+b_{o}) \tag{3}\]  
In the same manner as the hidden layer, \(f_{O}\) is the activation function and b is the bias vector.  
Although, this model maintains a memory of previous states, in practice it suffers from the vanishing gradient problem, thus becoming impractical for long-term dependencies [27]. A special type of RNN called Long Short Term Memory (LSTM) was published in 1997, which overcomes this issue [28]. LSTM cells follow a more sophisticated mechanism with the introduction of a complex cell that utilises \"forget\" gates to selectively choose what to forget. An illustration of the LSTM is given in Figure 2.  
Figure 1: Simple Recurrent Neural Network (RNN) architecture. For the sake of simplicity biases are ignored. Source: [26].  
The state of an LSTM memory unit adopts the following mathematical formulation:  
\[i_{t} =\sigma(W_{xi}+W_{hi}h_{t-1}+W_{ci}c_{t-1}+b_{i})\] \[f_{t} =\sigma(W_{xf}x_{t}+W_{hf}h_{t-1}+W_{cf}c_{t-1}+b_{f})\] \[c_{t} =f_{t}\otimes c_{t-1}+i_{t}\otimes tanh(W_{xc}x_{t}+W_{hc}h_{t-1}+ b_{c})\] \[o_{t} =\sigma(W_{xo}x_{t}+W_{ho}h_{t-1}+W_{co}c_{t}+b_{o})\] \[h_{t} =o_{t}\otimes tanh(c_{t})\]  
To clarify, the subscripts correspond to the initials of what each matrix represents (i.e., \(W_{hf}\) is the hidden forget weight matrix). Additionally, \(f\), \(i\), \(o\) and \(c\) correspond to the forget, input, output and cell gate vectors. With these in mind, an LSTM network would resemble the initial RNN architecture provided above but with LSTM cells instead.  
However, due to its architecture an LSTM network can only perform forward passes on sequential data, which ultimately means that the data dependencies are only modelled uni-directionally. An intuitive way to overcome this limitation is to use an exact replica of the LSTM network but in reverse. Thus, combining these two together, a Bidirectional LSTM (BiLSTM) is created which can be used to model dependencies bidirectionally.