# Transformer-Based Hybrid Forecasting Model for Multivariate Renewable Energy
## 3 H-Transformer
### Experimental Protocol  
The proposed hybrid system, named H-Transformer, was compared with two literature approaches in this work: single and hybrid. The single approaches selected were the traditional statistical linear model, SARIMA, and four state-of-the-art Machine Learning models: RNN [33], LSTM [34], GRU [35], and Transformer[24]. The codes for all experimentation and tests are available on GitHub ([[https://github.com/nyancy/h-transformer](https://github.com/nyancy/h-transformer)]([https://github.com/nyancy/h-transformer](https://github.com/nyancy/h-transformer)) (accessed on 23 October 2022)). The hybrid approaches were developed based on [18], as well as the proposed H-Transformer. This paper refers to these hybrid systems as SARIMA+RNN, SARIMA+LSTM, and SARIMA+GRU. The well-known Box and Jenkins methodology [36] was employed for selecting the hyperparameters \([p,d,q,P,D,Q,T]\) of the SARIMA model. This methodology is relevant because it guarantees that the linear patterns present in the time series be properly modeled. The BHO method was used to design the ML models of the single and hybrid approaches. The objective of the BHO is to find the best solution while avoiding local optima. The Bayesian algorithm builds a probabilistic model that maps the hyperparameter values and the loss evaluated on validation data when optimizing hyperparameters. The optimization iteratively evaluates and updates suitable configurations by balancing exploration (avoiding local minima) and exploitation (exploiting the optimum), using this information to find the optimum of the function. Because it uses a probabilistic model, Bayesian optimization has shown better results in the literature when compared to classic hyperparameter optimization algorithms such as random search and grid search [37]. The BHO tested 50 different combinations of hyperparameters for each ML model. Each combination ran for up to 300 epochs, stopping sooner if the validation error did not improve after 15 epochs (Early Stopping). The method created 200 combinations for each dataset and model, totaling 800 runs. The search only uses training and validation sets to evaluate its performance. The parameters and their values considered by the BHO for the RNN [33], LSTM [34], GRU [35] and Transformer [24] models were:  
* **units**, from 4 to 256;
* **learning_rate**, from 0.1 to 0.001;
* **batch_size**, from 8 to 256;
* **dropout**, from 0.1 to 0.4;
* **lags**, depends on the dataset. The Transformer model has a few additional parameters that are listed below. The hyperparameters considered on the original Transformer [24] constitute one of the possible configurations. * **head_size**, from 64 to 256;
* **head_number**, from 2 to 8;
* **block_number**, from 2 to 8;  
The loss used for optimization and comparison was the MSE (Mean Squared Error), computed by the Equation (7). The hybrid models used the SARIMA residual series as the target for the loss calculation. \[\text{MSE}=\frac{1}{n}\sum_{i=1}^{n}(Z_{t}-\hat{Z}_{t})^{2}, \tag{7}\]  
where \(Z_{t}\) and \(\hat{Z}_{t}\) are the actual series and forecasting at the time \(t\). Two evaluation metrics were used for performance comparison: Root Mean Squared Error (RMSE) and the Mean Absolute Error (MAE). The RMSE (Equation (8)) is an evaluation metric similar to the MSE. The square root is computed to transform the error into the same scale as the data. The MSE and RMSE are more sensitive to outliers when compared to the other metrics [38]. \[\text{RMSE}=\sqrt{\frac{1}{n}\sum_{i=1}^{n}(Z_{t}-\hat{Z}_{t})^{2}}. \tag{8}\]  
The CV(RMSE) (Coefficient of Variation of the RMSE) (Equation (9)) calculates the ratio of the standard deviation to the mean, dividing the obtained RMSE by the mean of the time series \(Z\) (\(\overline{Z}\)). It is used to indicate the amount of randomness between the data and the forecasted values. According to [39], a forecast model shall have a maximum CV(RMSE) of 25%, where lower values imply that the model outputs better forecasts.