# Benchmarking of Load Forecasting Methods Using Residential Smart Meter Data
## 3 Results
### In-Depth Analysis of the Trainted Models  
With the MARS, RF and ANN models already trained and providing the most accurate predictions for each consumer, it becomes relevant to proceed to a zoom-in look, with the intention of evaluating features' relevance, training times and also, for the case of ANNs, the networks' dimension. To evaluate features' relevance in MARS models, the dependence of each model on the different features was identified and, for the different hinge functions and eventual linear dependences of single features, an average of absolute coefficient values was computed. After aggregating the effect verified for the studied consumers, normalised averages (sum of values equal to 1) are presented in Figure 12. The labels are related with the used features (from x\({}_{0}\) to x\({}_{17}\)) already presented in Table 2. It can be noted that the features x\({}_{0}\), x\({}_{3}\), x\({}_{8}\), and x\({}_{13}\) tend to be the most influential in MARS models. These are related with past record values verified 8 days ago, 7 days ago and 2 days ago at the same hourly period being predicted. The strategy of including \"neighbouring\" records (not at the similar hourly periods, but at hourly periods hp \(-\) 2, hp \(-\) 1, hp \(+\) 1 and hp \(+\) 2) to give the effect of trend when forecasting, denotes specific features (such as x\({}_{2}\), x\({}_{14}\) and x\({}_{15}\)) to be interesting. In addition, the dependence on the cyclic variables (x\({}_{16}\) and x\({}_{17}\)) seems to be low. With a more detailed analysis, it can be concluded that for the individual models created for  
Figure 11: Second comparison of day-ahead predicted and measured load profiles (during a week of the test period)—#Consumer 59. Figure 12: Features’ relevance using MARS and RF models. each individual consumer, the contribution of these cyclic variables was imposing a linear dependence of these variables without the effect of hinge functions. As the input variables were not normalised, the ranges of these cyclic variables are considerably higher than the historical electricity consumption records (from \(x_{0}\) to \(x_{15}\)); thus, the resulting coefficients are considerably lower, distorting the real effect of these variables. Proceeding with the same analysis for the RF model and combining the RF features' relevance for the different models (individual consumers) through an average results in the normalised relevance of each feature, also shown in Figure 12.  
It can be validated that the electricity consumption values identified at similar hourly periods 7 days before and 2 days before the day being predicted are the most relevant in the forecasting process. In this case, the cyclic variable associated with the hourly period is also valuable, as the regression trees often create a branch to determine the following sub-node according to this feature. For the interpretation of relevant features in ANN models, after finding the best models, the weights obtained in the first layers (according to the connections between the different inputs and the different neurons assumed in each model) were analysed. For each ANN, an average of the absolute weights associated to each individual feature was assumed to characterise this corresponding feature. At the end, the relative importance of each feature was obtained by an average of its relative importance for the 71 consumers. The resulting normalised relevance for each individual feature is presented in Figure 13.  
Again, from the historical electricity consumption data, the electricity consumption records available at the similar hourly periods are the most important. In this case, the relative importance of adjacent hourly periods seems to be more relevant to pass to the ANN, to allow the perspective of transmitting the effect of trend. As before, the effect of day D-7 and day D-2 seems to be more important than that of day D-6. Finally, cyclic variables assume a relative importance in the context of ANN training.