## 2 Application of Neural Networks for Water Level Prediction
### Long Short-Term Memory Neural Networks (LSTM)  
The recurrent neural networks have at least a single feedback connection for their neurons, allowing for retaining previous information (i.e., memory) within the network [42]. However, deep RNNs suffer from vanishing and exploding gradient problems [43]. There are several variants of RNN existing, including LSTM, gated recurrent unit (GRU), and recurrent convolutional neural networks (RCNN) [44]. Among them, LSTM has excellent resistance to gradient problems [45].  
The memory capacity and controllability of the LSTM cell have improved by introducing gates to the LSTM cell. Gates supports NN in dealing with the problem of long-term dependencies. It consists of several types of gates within the cell, including forget gate, input gate, and output gate, as illustrated in Figure 2.  
Figure 1: A Multi-layer feed forward neural network.  
Figure 2: An illustration of LSTM cell.  
The mathematical model of an LSTM cell can be presented in the following Equations (2)-(7).  
\[\mathbf{f_{t}}=\sigma\ \big{(}W_{fh}\mathbf{h_{t-1}}+W_{\tilde{\mathbf{\ell}}}x_{t}+b_{ \tilde{\mathbf{\ell}}}\big{)} \tag{2}\]  
\[\mathbf{i_{t}}=\sigma\ \big{(}W_{ih}\mathbf{h_{t-1}}+W_{ix}\mathbf{x_{t}}+\mathbf{b_{i}}\big{)} \tag{3}\]  
\[\widetilde{\mathbf{c_{t}}}=\mathbf{tanh}\ \big{(}W_{\tilde{\mathbf{\ell}}}\mathbf{h_{t-1}}+W_{ \tilde{\mathbf{\ell}}}x_{t}+b_{\tilde{\mathbf{\ell}}}\big{)} \tag{4}\]  
\[\mathbf{c_{t}}=\mathbf{f_{t}}\mathbf{c_{t-1}}+\mathbf{i_{t}}\widetilde{\mathbf{c_{t}}} \tag{5}\]  
\[\mathbf{o_{t}}=\sigma\ \big{(}W_{\tilde{\mathbf{\ell}}}\mathbf{h_{t-1}}+W_{\alpha\mathbf{x_{t}}}+ \mathbf{b_{o}}\big{)} \tag{6}\]  
\[\mathbf{h_{t}}=\mathbf{o_{t}}\mathbf{tanh}(\mathbf{c_{t}}) \tag{7}\]  
where \(\mathbf{c_{t}}\) denotes the cell state and \(W_{i}\), \(W_{\tilde{\mathbf{\ell}}}\), \(W_{\mathbf{o}}\) represent the weights while the \"operator is for pointwise multiplication\". During the cell state update, the input gate decides what information to store in the cell; the output gate decides what information to send out; and the forget gate decides what information to erase from the cell during the next iteration.  
LSTM follows a slightly different algorithm than FFNN, named Back Propagation Through Time to converge the NN since it deals with temporal information related to previous timestamps. In addition, it has much better control of deciding what information to keep and what to remove in the next time stamp. The loss function for LSTM is presented in Equations (8) and (9).  
\[\mathbf{L_{\theta}}=\sum_{t}\mathbf{L_{\theta}^{(t)}} \tag{8}\]  
\[\mathbf{L(\theta)^{(t)}}=\frac{1}{2T}\sum_{i=1}^{T}\Big{(}\mathbf{d_{j}}-\mathbf{y(n)}_{ j}\Big{)}^{2} \tag{9}\]  
where, \(\mathbf{L_{\theta}^{(t)}}\) is the loss of time stamp \(\mathbf{t}\), \(\mathbf{L_{\theta}}\) is the cumulative loss, and \(\mathbf{T}\) is the total duration.  
Minimizing the computational load in terms of processing power and memory requirement for real-world water level prediction model implementation is essential, especially for rural and remote implementation. Depending on the NN type, hyperparameter combinations and the depth of the network will decide the computational load. However, in-depth comparisons will enable us to find simple NN models that serve powerful prediction performances compared to more complex NN models. In addition to NN models, genetic programming (GP), and deep GP models are more robust in prediction applications. However, to have a fair comparison between two types of relationships on common ground with low computational loading FFNN and LSTM NNs were selected for the study.