# LSTM Network for the Oxygen Concentration Modeling of a Wastewater Treatment Plant
## 3 Materials and Methods
### Long Short-Term Memory (LSTM) Network  
The LSTM [24; 25] network is a particular type of RNN characterized by a specific unit architecture, called a memory cell, that allows better learning of information dependencies compared with a simpler RNN. The LSTM memory cell presents an internal self-loop in addition to the outer feedback present in all RNNs, and a gating system that regulates the flow of information. In this way, the gradient flows over time, even for long periods, and its derivative remains computable (they neither vanish nor diverge). LSTM networks have been demonstrated to be able to dynamically map the input and output of a dynamical system; their use for learning the inherent properties of a given dynamical system has been studied in [14; 17; 18; 24; 26; 27; 28], obtaining promising results. They have proven to be powerful in representing long and short temporal dependencies in multiple examples with respect to other recurrent architectures, such as Gated Recurrent Unit (GRU), BIddirectional-LSTM (BI-LSTM), etc. [26]. For these reasons, the LSTM architecture was chosen as a promising candidate to model the process under study. The mathematical formulation of a single memory cell is represented in Figure 1 and can be expressed as follows:  
\[ \begin{split} c_{k}&=f_{k}\times c_{k-1}+i_{k}\times \tilde{c}_{k}\\ h_{k}&=o_{k}\times tanh(c_{k})\end{split}  \tag{1}\]  
where \(c_{k}\in\mathbb{R}^{n}\) is the internal cell state at time \(k\), \(u_{k}\in\mathbb{R}^{m}\) is the input at time \(k\), and \(h_{k}\in\mathbb{R}^{n}\) is the hidden state at time \(k\), with \(n\) the cell state dimension that corresponds to the number of LSTM neurons, \(m\) the number of features/input of the LSTM, \(\times\) is the Hadamard product and \(tanh\) is the hyperbolic tangent, used as an activation function. Four structures, called gates, regulate the internal loop of the cell, adding or removing information to modify the value of the cell state at each time instant. The first gate is the forget gate, \(f_{k}\); it decides which information is removed from the cell state \(c_{k}\), multiplying it by its past value \(c_{k-1}\), as can be seen in the first part of (1). Then, the input gate, \(i_{k}\), chooses which value of the cell state is updated, while the input activation gate, \(\tilde{c}_{k}\), creates a new candidate value of the cell state. The last gate is the output gate, \(o_{k}\), which determines which information contained in the cell state is included in output. The hidden state is then defined in (1), where the output gate is multiplied by a transformation of the cell state through a hyperbolic tangent function. These gates are expressed as:  
\[ \begin{split} f_{k}&=\sigma(W_{f}u_{k}+U_{f}h_{k-1} +b_{f})\\ i_{k}&=\sigma(W_{i}u_{k}+U_{i}h_{k-1}+b_{i})\\ \tilde{c}_{k}&=tanh(W_{c}u_{k}+U_{c}h_{k-1}+b_{c}) \\ o_{k}&=\sigma(W_{o}u_{k}+U_{o}h_{k-1}+b_{o}) \end{split}  \tag{2}\]  
where \(W_{f}\), \(W_{i}\), \(W_{o}\), \(W_{c}\in\mathbb{R}^{n\times m}\) are the input weight matrices, \(U_{f}\), \(U_{i}\), \(U_{o}\), \(U_{c}\in\mathbb{R}^{n\times n}\) are the previous output weight matrices, and \(b_{f}\), \(b_{i}\), \(b_{o}\), \(b_{c}\in\mathbb{R}^{n\times 1}\) are the biases. These quantities are specific to each gate and identified during the training of the network. The chosen activation functions are the hyperbolic tangent \(tanh\) and \(\sigma\), the sigmoid function, equal to:  
\[\sigma(x)=\frac{1}{1+e^{-x}}. \tag{3}\]  
Figure 1: Scheme of a single LSTM cell. Several LSTM cells are combined in the so-called LSTM layer, and the LSTM network can be composed of one or more stacked layers, representing the hidden layers.