## 3 Methodology
### Tuning the Forecasting Methods
#### 3.5.2 Deep Learning Methods Hyper-Parameters  
The architectural choices for all the neural network models have been designed as a trade-off between the possibility of representing more complex mapping functions (i.e., number of layers) and the opportunity of capturing the major possible number of mapping functions (i.e., number of nodes). Other than the network topology (number of layers and number of neurons/layers), we havebeen considering, the dimension of the input data window, the number of epochs, the batch size, the optimizer, the loss function, and the learning rate.  
We tested the input data window and the batch size which we set to 168 and 328 respectively. The reason we chose those two numbers is that the first contains the observations of one week while the second is a divisor of the training set. Through experimentation, we have seen that these two parameters give optimal results. Moreover, we monitored the training process in order to stop it at the point when performance on a validation dataset starts to degrade sensibly despite the tolerance parameter. We used the procedure of early stopping which is implemented within Keras API. Regarding the optimizer, we have been using Adam for MLP, CNN and LSTM while for the CNN-LSTM model we have been using SGD. To find the optimal learning rate for the CNN-LSTM model we use a technique that uses callbacks to tweak the learning rate using a learning rate scheduler so as to change the learning rates to a value based on the epoch number. By using this technique we are able to find the best learning rate which as shown in Figure 8a corresponds to \(10^{-5}\). We use this learning rate and obtain the graphics in Figure 8b which shows that the training process is stable from a learning point of view with the loss decreasing progressively even for several epochs larger than 200.  
* For the MLP neural network, we used a single hidden layer with 100 neurons. We analyzed the trend of the loss function with a different number of nodes from 100 to 900, observing no major differences. Regarding the number of hidden layers we empirically discovered that by adding more than one layer to MLP neural network we would just increase the time of training without benefits on the forecasting performance.
* For the CNN, we used a convolutional layer with 64 filters, a max-pooling with size equal to 2, followed by a dense layer with 50 neurons. Even in this case, we observed no significant changes with more complex architectures. As already said, the Adam optimizer was used during training.
* For the Encoder-Decoder LSTM, we used two layers containing 20 neurons each which we chose as a trade-off between simplicity and performance, similarly to what we did for MLP and CNN. As already said, the Adam optimizer was used during training.
* The CNN-LSTM architecture as shown in Figure 9 has one convolutional layer with 64 filters. Its output can then be interpreted by an LSTM decoder which contains 2 levels of 64 neurons each. Before the final output layer, two dense layers with 32 and 16 neurons, respectively, are used. For this model, we used the SGD optimizer and by experimenting with the learning rate as shown in Figure 8a we were able to pick the one that gives a stable decreasing loss as in Figure 8b. We have prepared a repository on Github with the CNN-LSTM ([[https://github.com/alketecaj12/cnn-lstm](https://github.com/alketecaj12/cnn-lstm)]([https://github.com/alketecaj12/cnn-lstm](https://github.com/alketecaj12/cnn-lstm))) model with data from one cell.  
Figure 8: (**a**) The loss function of the CNN-LSTM with varying learning rate and (**b**) the loss function with the best learning rate chosen in the previous step which decreases up to 200 epochs.