# Tuning ANN Hyperparameters for Forecasting Drinking Water Demand
## 3 Results and Discussion  
### Study of ANN Tuning Stochasticity  
As already discussed, ANN are a class of machine learning algorithms with an important stochastic component, which is necessary to improve the training of the algorithms. In addition, the high data variability that characterises the water demand makes the hyperparameters tuning complex. These two aspects lead to an increase of the tuning results variability, making the hyperparameters' selection a tricky task. In order to assess the stochasticity of ANN algorithms, 20 runs of the grid search tuning have been performed using the combinations of nodes and layers reported in Table 2. This analysis has been done on \(ts2\) for the 4 horizons with set of inputs 1 (Table 1) involving 2 ANN architectures with different characteristics: the FFNN and the LSTM architectures. This analysis set up has been taken due to the intensive computational requirements of this test. The results in terms of MAPE are reported in Figure 4 under Box and Whisker plots. Figure 4 highlights the variability that affects the simulations. In fact, each box, which represents a different configuration of nodes and layers of a defined problem and ANN architecture, shows a varying degree of variability of the simulations results by confirming the ANN stochasticity. This change of the results makes the choice of the best model more difficult and the tuning less reliable. It is also worth noting that the results of the 2 architectures have different behaviour. The LSTM shows overall better performance but also less variability compared to the FFNN. Besides, the FFNN resulting model with a simple structure is not properly able to deal with the more complex problem with large horizons like 24 and 168 h (panels (e) and (g)). This behaviour is even more underlined in Figure 5.  
Figure 5 shows the MAPE distribution of the models by aggregating the models based on the number of layers. This figure highlights, even more, the better performances achieved by the LSTM models compared to the FFNN. In addition, the variability trends that affect these models are well explained for the different ANN structures, consisting of 1, 2 and 3 layers. The variability of the models appears to significantly increase for the FFNN models along with the widening of prediction horizons. This behaviour affects the parameters tuning. The variability that affects the different solution makes the selection of the more performing configuration difficult. A solution for this issue can be to select the median, or some other statistical indicators, to compare the different configuration. In this study, the median is chosen. To underline the problem of selecting the more suitable neural network configuration, it is proposed to show how the rank changes with the increase of the number of simulations. The rank assesses with a number ranging from 1 to 24 which is the configuration with the lowest MAPE achieved by the median. In other words, it showsfor each iteration which is the configuration that is making the less MAPE with the median. This analysis is reported only for the horizon of 24 h in Figure 6.  
Figure 6 highlights the variability issue through the rank of the median results for each configuration updated at each iteration. Despite the rank keep changing during the simulations, some trends are stable and can be noticed in these plots. For instance, the panel (a) of Figure 6 shows the FFNN models for the 24 h horizon. In this case, the configuration with at least 2 layers have the better ranks, and consequently they are preferable compared to the single layers. In fact, as also depicted in panel e of Figure 2, the performance is significantly better both in terms of variability and also lower MAPE. With this analysis, it is possible to detect this behaviour and to select the more suitable configuration to deal with the variability ANN issue at least after the first 3 simulations. Regarding the LSTM model  
Figure 4: MAPE of the 20 simulations for each combination of nodes and layers of the _ts2_ with set of lags 1.