# Agnostic Energy Consumption Models for Heterogeneous GPUs in Cloud Computing
## 3 Power and Energy Models
### Model Design  
Some studies such as [26] have confirmed that the linear regression method is appropriate for old GPU architectures. Nevertheless, the study conducted in [16] has argued that linear models can not fit the complexity of modern GPUs. The relationship between the workload and power consumption in both GPUs tends to be non-linear. Therefore, DNN is selected to predict the power consumption for heterogeneous GPUs connected with VMs since DNN is a non-linear model and shows a better performance than other machine-learning algorithms [37]. Since every model's input has a different range of value scale, the collected data in each input attribute have been pre-processed by using the z-score technique since it is a popular technique [38]. The aim of using this technique is to simplify the procedure of model training. The z-score is represented by Equation (3):  
\[\text{Z}=\frac{\text{X}-\text{\mu}}{\sigma} \tag{3}\]where X is the element value--in this case, it will be the counter value for every application; \(\mu\) is the mean of all application values for a certain selected counter in the training set; and \(\sigma\) is the standard deviation of a single selected counter. We similarly use the process of \(\mu\) to calculate \(\sigma\). Then, we calculate \(Z\) for the remaining applications on other selected counters. After the inputs are pre-processed, they will be fed to the model's training stage. The model aims to develop a function f that maps the X inputs set to the Y outputs set f: \((\text{X}\rightarrow\text{Y})\) to estimate the power consumption P of a GPU (Fermi and Kepler) connected with a VM. Two power models are developed for the Fermi and Kepler GPUs. For the Fermi power model, the input layer has nine neurons representing every selected input, as shown in Table 3.  
After performing some experiments to find the adequate number of hidden layers and neuron numbers in every layer, we found that four hidden layers and eight neurons in each hidden layer are appropriate numbers. Therefore, the model contains four hidden layers to enable the layers to extract features from the data, and each hidden layer contains eight neurons; moreover, the output layer contains one neuron representing the predicted power consumption. For the Kepler GPU power model, the input layer consists of six neurons indicating the selected inputs for Kepler GPU. Like the Fermi Power model, the Kepler model has four hidden layers, and each hidden layer has eight neurons. Finally, the output layer contains one neuron to represent the predicted power consumption. Suppose \(x\in X=\{x_{1},\ x_{2},\ x_{3},\ \ldots,x_{n}\}\) is a selected input variable and \(w_{ij}\) represents a certain synapse weight S in each layer \(j\) which contains \(i\) neurons. Every \(s\) represents the summation of the product of all \(w_{ij}\) with \(x\) to every neuron by using Equation (4):  
\[s_{i}=\sum_{j}w_{ij}x_{i} \tag{4}\]  
After obtaining s for every neuron, the value will be passed to the activation function f(\(s_{i}\)) by using the appropriate non-linear function. In this model, the rectified linear unit (ReLU) function is used as an activation function. Since ReLU cannot be an activation function for the output layer, the activation function in the output layer \(s_{o}\) is the linear activation function. In DNN, to find the output of every layer \(l\) and map them together, Equation (5) can be used to estimate power consumption \(P_{d}\):  
\[P_{d}= \text{f}(W,X,b) \tag{5}\]  
After designing the model, the model training step is established. The loss function \(L\) is used to measure the difference between \(P_{d}\) and \(P_{ac}\) in each training step using Equation (6):  
\[L\left(p_{d}^{i},p_{ac}^{i}\right)=\frac{1}{2}\left(p_{d}^{i}-p_{ac}^{i}\right)^ {2} \tag{6}\]  
where \(i\) is a certain training step. The DNN training process is a time-consuming and hard task [43]. Therefore, the Adam optimization algorithm [44] is the selected algorithm for the model's training stage.