## 3 Results
### Comparing 4 Different ML Models  
Each input example is uniquely associated with one of the SkyImager pictures taken every 15 s. The normalized pixel values for the Red-Green-Blue (RGB) channels of an \(8\times 8\) resized subimage centered about the sun are flattened into one row vector. Note that other color spaces such as HSV or HSL could also be used. Resizing provided dimensionality reduction and reduced runtimes, but with substantial computer resources the 128 \(\times\) 128 sub-images could be used for training. Average values of each channel were included as additional features for a total of \(3\times 64+3=195\) features. The first entry in each row is the measured GHI in \(W/m^{2}\). To show how well random variables \(X\) and \(Y\) are correlated, one uses a scatter diagram. Figure 17 shows scatter diagrams of measured GHI versus predicted GHI for four ML models: Multi-Layer Perceptron (MLP), Random Forest (RF), Deep Learning (DL), and Gradient Boosted Trees (GBT). While all models perform well (tight clustering around \(y=x\), perfect correlation), DL and GBT have fewer outliers and visually outperform the other two models.  
Note that the MLP and RF models were run using the Scikit-Learn ML software package while DL and GBT were run on the Rapidminer platform. This validated our results on different ML packages--results should depend on the algorithms, not the platform on which they are implemented. Currently, Scikit-Learn does not offer a deep learning model. Using _Rapidminer_ is very convenient on a powerful desktop PC, our ultimate goal is to use the trained weights on a Raspberry Pi 3 computer for real time forecasting. Scikit-Learn with its open source Python interface should prove valuable for that task.  
MAE, MAPE, nRMSE, and \(R^{2}\) error metrics are given in Table 2 and run times in minutes. The explained variance (\(R^{2}\)) in the last column is very significant. Both DL and GBT achieve values of 0.87, while MLP and RF are 0.1 less. The other error metrics closely track the \(R^{2}\) values. In DL the extra accuracy is at the expense of much longer run times, but GBT gets the highest accuracy and is very fast: 10 min faster than RF.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**ML Model** & **Platform** & **Runtime (min)** & **MAE** & **MAPE** & **nRMSE** & \(R^{2}\) \\ \hline Multilayer Perceptron & Scikit-Learn & 3:05 & 81.21 & 33.05\% & 32\% & 0.71 \\ \hline Random Forest & Scikit-Learn & 14:53 & 66.86 & 28\% & 29\% & 0.76 \\ \hline Deep Learning & Rapidminer & 43:52 & 50.992 & 27.13\% & 21.6\% & 0.871 \\ \hline Gradient Boosted Trees & Rapidminer & 4:50 & 47.072 & 22.73\% & 21.1\% & 0.875 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Evaluation of four machine learning models.  
A time series is another approach to visualizing the results: GHI values are plotted on the \(y\)-axis and time on the \(x\)-axis. Figure 18 compares measured and predicted GHI for one day, 3 October 2015. Although there are differences in the two curves, they track each other well. More illuminating is a time series display for the entire group of testing days shown in Figure 19, where actual GHI is blue and forecast values are red. It is difficult to distinguish the two curves because they track each other so closely. For both figures the deep learning model was used for prediction. Observe in the center of Figure 19 a group of five consecutive clear sky days that are easy to predict, as are completely overcast days.  
The two curves differ most on moderately cloudy days with air mass cumulus clouds present.  
Figure 17: GHI actual vs GHI predicted on 24 October 2015 using different AI models.  
Figure 18: Time series of actual versus predicted GHI for single day of data.