# Benchmarking of Load Forecasting Methods Using Residential Smart Meter Data
## 3 Results
### Forecasting Models Comparison  
As mentioned earlier, for each one of the 71 consumers, the three different approaches (MARS, RF and ANN) were evaluated and compared with the _Native_ model. All the following presented results are related with the test sample. Attending to the _MAE_ metric, it can be stated that ANNs were the most accurate approach for 68 consumers and RF reveals the best performance for the remaining three consumers. In this case, despite MARS models showing similar forecasting performances compared with the concurrent ones, this approach was never the best one. With respect to the _MSE_ metric, ANN was the most accurate approach for 54 consumers, and RF was the most accurate for 16 consumers and the remaining consumer was better forecasted using the MARS model. The differences in the absolute error metrics are not so significant (as can be visualized in Figures 4 and 5); it is possible to infer that none of the different forecasting approaches applied somehow compromise an expected accuracy range. The radar charts allow a holistic view of the error analysis, with no prior logical order of the different consumers and highlighting the different forecasting methods. However, to allow a clearer analysis, bar charts were also used with an ascending trend of error metrics _MAE_ and _MSE_, using the MARS model as a reference in the ranking. The bar charts shown in Figures 6 and 7, due to the high number of consumers being involved, were split into two different graphs.  
Figure 4: Radar chart involving _MAE_ (in kWh) for different Consumer IDs.  
Figure 5: Radar chart involving _MSE_ (in kWh2) for different Consumer IDs.  
Figure 6: Bar charts involving _MAE_ (in kWh) in an ascending order.  
As mentioned earlier, scaled error metrics were also applied using the _Naive_ model as baseline. A more detailed comparison of different applied models is possible, as shown in Figures 8 and 9. As it is perceived, apart from some residual number of consumers, the forecasting methodologies are more precise than the _Naive_ model as expected (_MASE_ values are below the unitary value, highlighted as Reference in the chart). For most of the tested consumers, ANN is the most accurate model, but with no significant differences among the approaches. With this _MASE_ analysis, it can be concluded that RF models reveal some weaknesses when applied to some specific consumers. These low performances identified are often related with cases of overfitting, as will be further discussed.  
Regarding _MSSE_ analysis, it can be pointed out that for almost all consumers, the three proposed approaches considerably reduce the error metrics identified in the _Naive_ model (_MSSE_ typically below 0.8).  
With the perspective of conveying the challenge associated with domestic load forecasting, Figures 10 and 11 present measured weekly load diagrams during the test period and the short-term predictions (day-ahead forecasting) available by using the proposed methodologies. It must be stressed that the electricity consumption records available in the time series were converted to active power to build up these load profiles.  
The randomness in resulting active power series is clear and some abrupt changes (sudden peak values or unexpected low values) in active power load are in some cases difficult to predict with the different methodologies. The two presented consumers were chosen due to considerably different consumption volumes as well as load patterns.  
Figure 7: Bar charts involving _MSE_ (in kWh2) in an ascending order.  
Figure 8: _MASE_ comparison per consumer ID.  
Figure 10: First comparison of day-ahead predicted and measured load diagrams (during a week of the test period)—#Consumer 18.  
Figure 9: _MSSE_ comparison per consumer ID.