# Tuning ANN Hyperparameters for Forecasting Drinking Water Demand
## 2 Methods and Materials
### Methods
#### 2.2.2 ANN Architectures  
This section aims to present the four neural network architectures adopted in this investigation. The proposed models have been developed with the KERAS [32] and TensorFlow [33] Python frameworks.  
Feed Forward Neural Network  
The first proposed architecture is probably the most classical and used one, known also as multilayer perceptron. These networks are based on a succession of units, called neurons, that are grouped in a sequence of layers. The first called the input layer, whose purpose is to connect the input information to the subsequent layers, which are the hidden ones. In this case, it is possible to have one or more hidden layers, producing in this way a deep neural network. Lastly, the output layer provides the outcome of the model. Due to its relative simplicity, this class of network has been widely adopted in many studies [4] and has become a benchmark for the forecasting process.  
The simple recurrent neural network (SRNN) is a class of neural networks whose purpose is to remember information during the time. For this reason, their connections are different from the multilayer perceptron and they are designed to have not only feedforward connections but also feedback connections. Despite that this feature should allow the SRNN to remember past information with long term dependencies, some studies showed that this class of network was actually not capable to remember, and the reason is the vanishing gradient problem [34]. To overcome this issue, a different variation of recurrent neural networks has been proposed, like the LSTM [35] or the GRU [36].  
Long Short-Term Memory  
LSTM has been designed to preserve long term information, avoiding the vanishing gradient problem. These particular networks have each computing units organised in a series of gates aimed at the management of the memory. These include the forget, the input, the cell and the output gates. With this complex sequence of individual hidden units, the LSTM structure has to learn how to store information, how to remember information and what output to give. It is worth noting that this is a simplified explanation of the complex mechanism that is behind the LSTMs. There exists a plethora of studies developed in recent years that deal with this architecture, for instance [37].  
Gated Recurrent Unit  
The LSTM networks were proposed in 1997 to solve the vanishing gradient problem and perform in language processing problems [35]. Despite their effectiveness and their ability to deal with long-term information, these networks have a complex structure that makes their training much slower compared to conventional neural networks. To make the process faster, the GRU networks were proposed as a modification of LSTM networks. Similarly to the LSTM, the GRU has to capture information with dependencies at long time scales, with the difference that it is simpler to compute and easier to implement [36]. The GRU network has the same gate mechanism as the LSTM with some differences. The most important is the combination of the input and of the forget gates into a single one and the lack of a distinct cell state.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Layers** & **Nodes** \\ \hline
1 & [18], [36], [48], [72], [96] \\
2 & [18-18], [18-36], [36], [36], [36], [36