# A System for Sustainable Usage of Computing Resources Leveraging Deep Learning Predictions
## 3 System Setup and Deep Learning Algorithms
### Deep Learning Algorithms  
The paper offers a comparative study of three popular deep neural network architectures and validates their implementations in Tensorflow [37] on the collected CPU metrics. The following types of deep neural networks are used in the implementation of the algorithms:  
* Convolutional neural network (CNN) model--3 layers of 1-dimensional convolutions followed by a batch normalization layer and a rectified linear unit layer to apply non-linear transformation to the data. The resulting tensors are passed through a last pooling layer that averages the results of the convolutions and is fed into a 1-neuron dense layer that outputs a predicted value. * Recurring neural network (RNN) model--2 layers of a simple recurring neural network are used in order to train the model, the first layer feeding a sequence into the second one, then passing the information to a fully-connected dense neural network layer. * Long short-term memory (LSTM) model--2 layers of an LSTM based neural network, using specialized recurring neural network cells that have a special memory layer feeding into each of the neurons in the sequence. The layers both return sequences and feed into another pair of dense neural network layers before outputting a prediction. The loss function used in order to perform the stochastic gradient descent optimization for each of the models is the Huber loss function, which has a history of being adapted to time-series and regression problems [38]. Additionally, in order to find the optimal learning rate in order to avoid over or under-fitting the model, a learning rate scheduler is used for each algorithm for 100 epochs. The output of the scheduler seen on a graph shows the correct moment in which the loss on the training set starts increasing due to a too-high learning rate. Figures 5-7 show the output in terms of mean absolute error and loss for the models, at each epoch. As it gets higher, the loss starts increasing and it is essential to pick the learning rate that was found right before the loss begins decaying. Figure 5: Learning rate schedule output—CNN. Figure 6: Learning rate schedule output—LSTM. Figure 7: Learning rate schedule output—RNN. The rest of the parameters, such as those used to establish the batch sizes and the window of data taken into consideration, were selected based on practicality. A batch of 32 windows in the data are taken for each training epoch, while the window size is equal to 20. We took 20 as it represents 100 min per input to the algorithm, 1 h and 20 min, a realistic value to be used when doing online prediction in the future. Table 1 holds a breakdown of the number of parameters available for training each of the models, as well as the optimal learning rate that was found. The optimal learning rate can be obtained using the built-in LearningRateScheduler class provided in the Keras implementation of Tensorflow 2. Using stochastic gradient descent with an initial low learning rate \(10^{-8}\) and a momentum of 0.9, Figures 5-7 show the results of training for an epoch with each step of the parameter. The higher the learning rate, the quicker the algorithm converges. However, fast convergence leads to potentially missing the global minimum and \"bouncing around\" the target, eventually increasing the loss. The values presented in Table 1 are the ones just before the loss starts increasing, leading to the conclusion that they are the parameters for which conversion occurs the fastest. The number of parameters comes not by choice but from the design of each of the networks. The trainable parameters of the models are further described, layer by layer, in Tables 2-4. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Architecture** & **No. Parameters** & **Optimal LR** \\ \hline CNN & 25,793 & \(1.6\times 10^{-5}\) \\ LSTM + DNN & 17,573 & \(1\times 10^{-5}\) \\ RNN & 4961 & \(7\times 10^{-6}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Number of parameters and optimal learning rate for each architecture.