# A State-of-Art-Review on Machine-Learning Based Methods for PV
## 2 Machine Learning, Deep Learning and Related Methods
Usually, main performance gains in models' predictive performance are possible by performing \"feature engineering\", i.e., combining raw features into new features that can express new/more knowledge on the system to which the data set is related [17]. This \"feature engineering\" or \"feature extraction\" that has to be manually implemented in SNNs is automatically performed in DNNs, at least to some extent. Another ML-based method that is beginning to be employed in the field of PV, especially for MPPT reactive tracking, is reinforcement learning (RL). While in \"traditional\" ML/DL methods a dataset is required to extract knowledge from the data (training phase) and thereafter apply this knowledge to new unseen data (inference phase), in RL the model, or better, the system, can learn by themselves essentially by trial and error. Using RL an \"agent\" performs actions to maximize rewards, or in other words, it is learning by doing, and its goal is to optimize the total reward in the same way as ML or DL aim at minimizing a loss function. In addition to DL, during the last several decades, a class of methods known as Ensemble Methods (EM) has been developed and has started to appear in research papers [18; 19; 20]. The basic idea is quite simple: integrate a group of base models, also known as weak learners, to build up a more robust model. This robustness is intended to build a model capable of providing better accuracy, performing better, and/or being capable of better generalizing, i.e., to provide good performance for a \"scenario\" different from the training one. However, how does one train different weak learners and aggregate their output to build up a stronger leaner? In this regard many solutions are possible, but commonly used techniques are:  
1. Bagging
2. Boosting
3. Stacking  
Bagging stands for Bootstrap Aggregation, where multiple models are trained in parallel, but each base model is trained on a different training set derived from the original training data using the Boostrap (data is randomly sampled from the original dataset with replacement) method and the final prediction is derived by a voting aggregation from the predictions of all base models. In bagging methods, the weak learners are usually of the same type. Since the random sampling with replacement creates independent and identically distributed samples, bagging does not change the models' biases but reduces their variance, producing a model capable of providing consistent results in production. A typical bagging model is based upon Random Forest. In boosting, multiple weak learners are learned sequentially, not in parallel as in bagging. Each subsequent model is trained by giving more importance to the data points that were misclassified (or giving greater error in terms of MSE for example) by the previous weak learner. In this way, the weak learners can focus on specific data points and can collectively reduce the bias of the prediction. In stacking, the base weak learners are trained in parallel as in bagging, but stacking does not carry out simple voting to aggregate the output of each weak learner to calculate the final prediction. Stacking employs another meta-learner to provide the final prediction, and this meta-learner is trained on the outputs of weak-learners to learn a mapping from the weak learners' output to the final prediction. Usually, this meta-learner is quite simple, such as a LASSO or Ridge regression. Previously, the terms bias and variance have been cited a few times, and require further clarification. A common \"mantra\" in ML is the bias vs. variance trade-off; any ML-based model trying to improve bias will always make gains at the expense of variance, and vice versa. The two variables measure the effectiveness of the model: bias is the error or difference between real data and a models' predicted value, while variance is the error that occurs due to sensitivity to small changes in the training set.