# ForecastNet Wind Power Prediction Based on Spatio-Temporal Distribution
## 2 Materials and Methods
### Wind Power Prediction Model
#### 2.2.3 ForecastNet Model  
ForecastNet [43] is a multi-layer feed-forward neural network model that is commonly used to perform multi-step time series forecasting. Its network structure is shown in Figure 4, where each neuron can receive signals from the previous layer of neurons, resulting in a multilayer structure. This neural network structure is commonly used for processing time series data because it captures trends and periodicities in time series, as well as other complex dynamic features. ForecastNet has time-varying and interleaved output properties among model neurons. The former improves the problem of gradient vanishing during the training process of recurrent neural network (RNN) and CNN models. The latter improves the problem of gradient explosion and gradient vanishing that occurs during neural network training. The root cause of gradient explosion and gradient vanishing is the reuse of chaining laws in neural network gradient computation. The interleaved output properties of ForecastNet are shown in Figure 5, where the upper row represents the hidden layer network, the lower row represents the output layer, \(a^{[i]}\) is the output vector of the \(l\)-th layer, \(z^{[l]}=W^{[l]}T^{a[l-1]}b^{[l]}\), \(W^{[l]}\) is the weight matrix of the \(l\)-th layer, and \(b^{[l]}\) is the bias parameter matrix. It breaks down the chain multiplication chain in the chain rule into the sum of multiple terms. The factor accumulation process is more stable than the multiplication process, which can significantly reduce the depth of the network while alleviating the gradient explosion and gradient vanishing problems. Figure 4: Multilayer feedforward neural network structure. Figure 5: The interleaved output properties of ForecastNet. The input layer of ForecastNet is univariate or a set of multivariate inputs; the hidden layers are different forms of feed-forward neural networks, such as common back propagation (BP) networks, radial basis function (RBF) networks, etc. The architecture of each hidden layer can be heterogeneous or identical, and the parameters of each hidden layer are independent of each other, which are used to simulate the dynamic characteristics of the time series. Different variants of the ForecastNet model can be obtained by using different feed-forward networks in the hidden layer:  
6. ForecastNet model with a multilayer perceptron (MLP) as the hidden layer (MLPFN)  
A multilayer perceptron is a special form of a fully connected neural network. Its main difference from a fully connected network is its hidden layer. The hidden layer can improve the MLP's expressive ability for the network, thus improving its ability to solve complex prediction or classification problems. Figure 6 shows a schematic diagram of the ForecastNet hidden layer using an MLP structure, where dense is the fully connected layer and \(h\) represents the number of neuron nodes in each hidden layer. The hidden layer and the output layer of the MLP network are fully connected layers. Each layer has 24 ReLU neuron units, where the neuron nodes are fully connected. 7. ForecastNet model with CNN as the hidden layer (CNNFN)  
CNN is a kind of artificial neural network. Its structure is mainly composed of three parts: the convolution layer, pooling layer, and dense layer, as shown in Figure 7. The main role of the convolution layer is to extract the features, where \(f\) is the number of convolution kernels and \(k\) is the size of the convolution kernel. The pooling layer is used for down sampling, where \(s\) is the filling of the pooling layer and \(p\) is the step size of the pooling layer. The dense layer is mainly used for feature classification, where \(h\) is the number of hidden layers of the dense layer, which is composed of 24 ReLU neurons. 8. ForecastNet model with the hidden layer as a dense network based on the attention mechanism (ADFN) The model is shown in Figure 8.  
Figure 6: Hidden layer structure of MLPFN. Figure 7: Hidden layer structure of CNNFN.