## 3 Popular ANN Architectures Employed for Time Series Forecasting Purposes
### Self-Organizing Maps
#### 3.5.1 Essential Characteristics and Training of an SOM  
An SOM is organized as follows: we have points \(\vec{x}\) in the input space that are mapped to points \(I(\vec{x})\) in the output space. There is a set of points \(\vec{x}\) living in the input space, and we suppose that there is a function to assign \(\vec{x}\) to points \(I(\vec{x})\) in the output space. In turn, there is another function to assign to each point \(I\) in the output space a corresponding point \(\vec{w}(I)\) in the input space (see Figure 4). Figure 4: Organization of an SOM. Kohonen networks [60] are a particular and important kind of SOM. The proposed network has a feedforward structure with a single computational layer, where the neurons are arranged in rows and columns. Nodes in the input layer connect to each of the neurons in the computational layer (see Figure 5). \[d_{j}(\vec{x})=\sum_{i=1}^{D}(x_{i}-w_{ji})^{2} \tag{8}\]  
The self-organization process consists of the following components:  
1. _Initialization_. At first, the connection weights are set to small random values. 2. _Competition_. For a \(D\) dimensionality input space, \(\vec{x}=\langle x_{1},\dots,x_{D}\rangle\) represents the input patterns and \(\vec{w}_{j}=\langle w_{j1},\dots,w_{jD}\rangle\) represents the connection weights between the input units \(x_{i}\) and the neuron \(j\) in the computational layer; \(j=1,\dots,N\), where \(N\) is the total number of neurons. The difference \(\vec{x}\) between \(\vec{w}_{j}\) for each neuron can be calculated as the Euclidean distance squared, which will represent the discriminant function \(d(\vec{x})\). The neuron with the lowest discriminant function \(d(\vec{x})\) is declared the winner-takes-all neuron. Competition between neurons allows mapping the continuous input space to the discrete output space. 3. _Cooperation_. In neurobiological studies, it was observed that, within a set of excited neurons, there can be lateral interaction. When a neuron is activated, the neurons in its surroundings tend to become more excited than those further away. A similar topological neighborhood that decays with distance exists for neurons in an SOM. Let \(S_{ij}\) be the lateral distance between any pair of neurons \(i\) and \(j\), then \(T_{j,l(\vec{x})}=\exp(-S_{j,l(\text{ }n\text{ }excx)}^{2}/2\sigma^{2})\) defines our topological neighborhood, where \(l(\vec{x})\) is the index of the winner-takes-all neuron. A special quality of the SOM is that the size of the neighborhood \(\sigma\) should decrease over time. An exponential reduction is a commonly used time dependence: \(\sigma(t)=\sigma_{0}\exp(-t/\tau_{\sigma})\). 4. _Adaptation_. SOM has an adaptive (learning) process through which the feature map between inputs and outputs is formed through the self-organization of the latter. Due to the topographic neighborhood, when the weights of the winner-takes-all neuron are updated, the weights of its neighbors are also updated, although to a lesser extent. To update the weight, we define \(\Delta w_{ji}=\eta(t)\cdot T_{j,l(\vec{x})}(t)\cdot(x_{i}-w_{ji})\), in which we have a time-dependent learning rate \(t\)\(\eta(t)=\eta_{0}exp(-t/\tau_{\eta})\). These updates are applied to all training patterns \(\vec{x}\) for various periods. The goal of each learning weight update is to move the weight vectors \(\vec{w}_{j}\) of the winner-takes-all neuron and its neighbors closer to the input vector \(\vec{x}\). When the SOM training algorithm has converged, important statistical properties of the feature map are displayed. As shown in Figure 4, the set of weight vectors \(\{\vec{w}_{j}\}\) in the output space integrates the feature map \(\Phi\), which provides an approximation to  
Figure 5: Kohonen network. the input space.