## 3 Self-Attention Neural Network
Once the self-attention mechanism network architecture has been defined, it is necessary to empirically analyze it and determine the value of hyperparameters that maximize the result (lower MAE or RMSE) with the lowest computational cost. Statistical analysis allows only considering relevant factors, which result in a simpler model to train. Python (v. 3.11.12) and specifically its Keras (v. 3.8.0) and Tensorflow (v. 2.18.0) libraries have been used to design the neural network, which are intended for the creation of deep learning models and networks. They also allow great flexibility by allowing user-customizable configurations. The Concepts and Procedures for the Utilization of Design of Experiments (DoE), ARIMAX, and Bootstrap  
Most practical situations where it is intended to control the output or response of a real process are conditioned by the existence of controllable variables (factors) and other unknown factors. The DoE is a methodology proposed by [PERSON] [68] and it encompasses a set of mathematical and statistical techniques that seek the effects on a measured response variable (Y) by simultaneously manipulating certain input variables (Xs). In the application of DoE techniques, the main guidelines may include the following: (1) stating the objectives of the research, (2) defining the response variable(s), (3) determining factors and levels, (4) deciding the experimental design type, (5) conducting the experiment, (6) data analysis, and (7) validation of the results. With the intention of estimating first and second order effects and cross-product, two-level factorial designs are used, and the polynomial function must contain quadratic terms. An orthogonal design is shown in the equation below:  
\[y=\beta_{0}+\sum_{i=1}^{k}\beta_{i}x_{i}+\sum_{i=1}^{k}\beta_{ii}x_{i}^{2}+ \sum_{1\leq i<j}^{k}\beta_{ij}x_{i}x_{j}+\varepsilon \tag{2}\]  
where \(k\) is the number of variables, \(\beta_{0}\) is the constant term, \(\beta_{i}\) represents the coefficients of the linear parameters, \(x_{i}\) represents the variables, \(\beta_{ii}\) represents the coefficients of the quadratic parameters, \(\beta_{ij}\) represents the cross-product, and \(\varepsilon\) is the residual associated with the experiment (random error). The variables are usually expressed in different units and vary in different ranges, which makes it necessary to code them. If this is not accomplished, it is impossible to assess the relevance of a factor through the magnitude of the model coefficients. As a result, all factor levels are set at two levels (\(-1\), \(+1\)) with equal spaced intervals between them. Hence, the factorial experiments with multiple factors with two levels (\"high\" and \"low\"), suggest that the complexity of the model grows exponentially (\(2^{k}\)), and the experiment requires a lot of resources and time. Since a low number of main effects and lower order interactions are significant to the response variable, and higher order interactions are not significant, a fractional factorial design may be chosen. As an extension of the widely used ARIMA (autoregressive integrated moving average) forecasting model, the ARIMAX allows for the inclusion of one or more exogenous variables, thereby enabling more satisfactory predictions. The model can be expressed as follows:  
\[\big{(}1-\phi_{1}L-\ldots-\phi_{p}L^{p}\big{)}(1-L)^{d}y_{t}=c+\beta X_{t}+(1 +\theta_{1}L+\ldots+\theta_{q}L^{d})\epsilon_{t} \tag{3}\]  
where \(y_{t}\) is the value of the time series at time t, \(X_{t}\) is the exogenous variable at time t, \(\beta\) is the coefficient that measures the impact of the exogenous variable, \(\phi_{1},\ldots,\phi_{p}\) are autoregressive coefficients, \(\theta_{1},\ldots,\theta_{q}\) are moving average coefficients, c is de average level of the time series when all other variables are zero (AR(p), MA(q), \(\beta\)), L in the lag operator, and finally, \(\epsilon_{t}\) is the error term at time t. The proper use of this model requires the time series to be stationary, i.e., it must have constant mean and variance over time.