## 4 Brief Literature Survey on ANNs Applied to Time Series Modeling
### Combining Feedforward Neural Networks and Particle Swarm Optimization for Time Series Forecasting
#### 4.1.2 Particle Swarm Optimization Convergence
In their study, [PERSON], [PERSON], and [PERSON] (2018) propose an empirical comparison between three different models:  
1. FFNNs;
2. SVR models using a radial basis function (RBF) kernel;
3. SVR models using a linear function kernel. This study reflects, for the considered models, their ability to learn and forecast fundamental time series patterns relevant to empirical forecasting tasks. Next, SVR models are briefly described (description based on text in [PERSON], [PERSON], and [PERSON] (2018)). **Support vector regression**. For their experiment, [PERSON], [PERSON], and [PERSON] (2018) employed the common support vector regression (SVR) algorithm described in ([PERSON] et al., 2018), which applies an \(\epsilon\)-insensitive loss function for predictive regression problems. Let \(\{(\vec{x}_{i},y_{1}),\ldots,(\vec{x}_{n},y_{n})\}\), where \(\vec{x}_{i}\in\mathbb{R}^{d}\) and \(y_{i}\in\mathbb{R}\) are the training data points available to build a regression model. A transformation function \(\Phi\) on the initial input space is applied to map the original data points to a higher dimensional feature space  
Here, \(\alpha_{i}\) and \(\alpha_{i}^{*}\) are the dual variables, and the expression \(K(\vec{x}_{i},\vec{x})\) represents the inner product between \(\Phi(\vec{x}_{i})\) and \(\Phi(\vec{x})\), which is known as the kernel function. It is possible to achieve a solution to the original regression problem, starting from the existence of the kernel function, leaving aside the transformation \(\Phi(\vec{x})\) applied to the data. For more information on SVR models, please consult [PERSON], [PERSON], [PERSON], [PERSON], and [PERSON] (2018). **Experiments and results**. [PERSON], [PERSON], and [PERSON] (2019) employed a set of five artificial time series in their experiments (see Figure 6):  
1. Stationarytime series (constant level);
2. Stationary time series with additive seasonality;
3. Linear trend;
4. Linear trend with additive seasonality;
5. Linear trend with multiplicative seasonality. These artificial data emulate the behavior of monthly retail sales and are taken from [PERSON]'s original classification. All artificial series contain additive Gaussian white noise, with \(\sigma^{2}=25\). Each time series consists of 228 observations. A lag structure of 13 previous observations was established to produce one-step-ahead forecasts (this number of lags should be adequate to capture seasonal patterns present in monthly time series). Thus, 215 examples are available to construct FFNN and SVR models. From these available examples, the first 119 were reserved for model training, the next 48 for model validation, and the last 48 for model testing. All models were constructed by using training and validation data only, retaining all information in the test suite to ensure valid predicted event testing. To avoid saturation effects, a linear scale in an interval \([-0.5,0.5]\) was applied in the data transformation, using minimum and maximum values only from the training and validation data. In order to evaluate model performance, mean squared error (MSE), mean absolute error (MAE), and root mean squared error (RMSE) metrics were employed to measure test errors. MAE was employed to fine-tune model meta-parameters (\(C\) and \(\epsilon\) for SVR-linear; \(C\), \(\epsilon\), and \(\sigma\) for SVR-RBF; number of hidden nodes and type of activation function for FFNN hidden nodes; available options were sigmoid or hyperbolic tangent). After the construction of all models and test error measuring, [PERSON], [PERSON], and [PERSON] (2019) arrived at the following conclusions:  
Figure 6: Some basic time series patterns according to [PERSON]’s classification. All of these patterns (except for the no trend + multiplicative seasonality) were generated artificially in the experiment of [PERSON], [PERSON], and [PERSON] (2019).