# Machine Learning and Deep Learning Models Applied to Photovoltaic Production Forecasting
## 2 Application and Assessment of Models
### Modeling
#### 2.2.2 Optimization Carried out by Parameters Adjustment  
The general purpose of this models is to have the minimum error that these have in the forecast of the objective variable. The mean squared error is selected as the loss function, \(l(y_{k},\hat{y}_{k})\), to assess the accuracy of the models, Equation (2). \[l(y_{k},\hat{y}_{k})=\sum_{k=1}^{n}\frac{\left(\hat{y}_{k}-y_{k}\right)^{2}}{n} \tag{2}\]  
where, \(n\) is the number of samples, \(\hat{y}_{k}\) is the forecast variable at time \(k\), and \(y_{k}\) is the objective variable at time k.  
* Decision trees RF model consists of the synchronized operation of several unrelated decision trees. Training process is made through bagging. This technique ensures the independence of each of the decision trees and reduces the sensibility of the model to data variations [51; 52]. The cost function of this model, \(J\), depends on the complexity of each of the decision trees, \(\omega(f_{t})\), in addition to the loss function, Equation (3). \[J=\sum_{k=1}^{n}l(y_{k},\hat{y}_{k})+\sum_{j=1}^{t}\omega(f_{j})\] (3) where \(t\) refers to the evaluated decision tree. The final result is weighted by the number of decision trees. Figure 3 shows a representative version of the random forest models. Figure 2: Configuration of a Support Vector Regression model. The main problem with the RF model is that it has decision trees without a good forecast, known as weak learners [53]. XGBoost model face this issue through additive learning. With this technique, the learning process considers time steps prior to forecasting the current value. This generates a modification in the cost function, which is commonly represented as a second-order expansion of the Taylor series, Equation (4). Thus, it is possible to obtain an accurate optimization of the problem quickly [54]. \[J^{(it)}=\sum_{k=1}^{n}\left[l\left(y_{k},\;\hat{y}_{k}^{(it-1)}\right)+g_{k} \cdot f_{t}(x_{k})+\frac{1}{2}h_{k}\cdot f_{t}^{2}(x_{k})\right]+\omega(f_{t}) \tag{4}\]  
where, \(it\) is the considered iteration, \(f_{t}(x_{k})\) is the definition of the \(t\)-tree, \(g_{k}\) is the first partial derivative of the loss function with respect to \(\hat{y}_{k}\), and \(h_{k}\) is the second partial derivative of the loss function with respect to \(\hat{y}_{k}\). Figure 3 also captures the decomposition of this model. * Neural Networks  
Deep Learning models have been configured using the bottleneck composition, where the number of hidden neurons decreases as the number of hidden layers increases [55]. Patterns are extracted by means of the interconnection of the different sets of neurons. The Deep Learning models propagate information from the input layer to the output layer through these hidden interconnections [56]. The cost function in these models is directly governed by the loss function, Equation (5). \[J=l(y_{k},\hat{y}_{k}) \tag{5}\]  
The SNN model is the basic configuration. In this model, neurons in one layer are fully connected to the previous layer. This type of layer is also known as dense layer. The other configurations, i.e., RNN model and CNN model, are commonly applied together with the SNN model. Thus, both models are used as the first hidden layers to extract important information. Then, the model changes and has several dense layers, typical of SNN modeling. The typical configuration of Deep Learning models is shown in Figure 4.  
The RNN model is a derivative of the SNN model that retains the important information from previous time steps, i.e., it has memory. LSTM (Long Short-Term Memory) is the best known type in the RNN modeling. This presents a series of internal mechanisms, gates, that regulate the flux of information that must be retained or forgotten [57; 58]. Due to the memory fact, the RNN model is more complex than the SNN model as it has more parameters to train. Figure 3: Configuration of a Random Forest model. The CNN model is also a derivative of the SNN model. This keeps important details and avoids irrelevant information.