# Short-Term Wind Speed Prediction Based on Principal Component Analysis and LSTM
## 2 Methodology
### LSTM and Hyperparameter Selection
#### 2.2.2 Selection of Hyperparameters  
Many parameters for LSTM can affect its accuracy and performance. The selected hyperparameters are a learning rate, the number of units of the hidden layer, and the number of batch size. If the selected learning rate is too small, the convergence will be too slow; otherwise, the cost function will oscillate. The number of units of the hidden layer will influence the effect of fitting. For the number of batch size, if this number is too small, then the training data will be extremely difficult to converge, which will lead to underfitting. If the number is too large, then the required memory will increase significantly. For example, when the number of units of the hidden layer is specified within (1, 100) and the range of the number of batch size is specified within (1, 500), a total of 50,000 combinations will be generated. Thus, to overcome the computational burden, a simple yet reliable algorithm should be utilized to select the optimal combination of parameters for balancing predictive performance and computational efficiency. In this paper, the hyperparameters of LSTM are determined by means of the DE algorithm. DE is a heuristic random search algorithm based on group differences [40]. The objective function of the optimization problem to select hyperparameters is the root mean square error (RMSE), which represents the sum of the squared deviations of the predicted value and the true value calculated by Equation (19):  
\[RMSE=\sqrt{\frac{1}{s}\sum_{i=1}^{s}\left(\tilde{y}_{i}-y_{i}\right)^{2}} \tag{19}\]  
where \(\tilde{y}_{s}\) is the \(s\)th predicted value, \(y_{s}\) is the \(s\)th true value.  
The process of the DE algorithm to select LSTM hyperparameters is shown below:  
1. Initialization: Initialize the following parameters: length of individual D, number of iterations G, population size NP, crossover rate, and scaling or mutation factor. The population is randomly generated by Equation (20): \[X_{\omega,k}=X_{\omega,k}^{L}+rand*(X_{\omega,k}^{U}-X_{\omega,k}^{L})\] (20) where \(\omega\) = 1, 2,..., NP; \(k\) = 1, 2,... D; \(X_{\omega,k}^{L}\) and \(X_{\omega,k}^{U}\) are the upper and lower bounds of the \(k\)-th dimension, respectively.
2. Mutation: Mutation operator is used to generate the mutation vector (\(H_{i}\)) for the individual of the population using Equation (21). \[H_{i}(g+1)=X_{r1}(g)+F*(X_{r2}(g)-X_{r3}(g))\] (21) where \(x_{r1}(g)\), \(x_{r2}(g)\), \(x_{r3}(g)\) are individuals randomly selected from the population, and \(r1\
eq r2\
eq r3\), \(F\) is the scaling factor and \(g\) represents the g-th generation.
3. Crossover: Crossover operation is to randomly select individuals using Equation (22) \[U_{\omega,k}(g+1)=\left\{\begin{array}{cc}H_{\omega,k}(g+1)&if\ rand(0,1) \leq CR\\ X_{\omega,k}(g)&otherwise\end{array}\right.\] (22) where \(U_{\omega,k}\) is the new individual generated in the crossover operation, \(CR\) is the crossover rate.
4. Selection: In the selection operator as shown in Equation (23), for minimization problems, if the fitness value \(f(U_{\omega,k}(g+1))\) of the trial vector \(U_{\omega,k}(g+1)\) is less than or equal to the fitness value \(f(X_{\omega,k}(g))\) of the target vector \(X_{\omega,k}(g)\), the trial vector will replace the target vector and enter the population. Otherwise, the target vector is still retained.  
\[X_{\omega,k}(g+1)=\begin{cases}U_{\omega,k}(g+1)&if\;f(U_{\omega,k}(g+1)\leq f (X_{\omega,k}(g))\\ &X_{\omega,k}(g)\end{cases} \tag{23}\]  
The best individual is the three hyperparameters of the LSTM prediction network structure. In this paper, the best set of hyperparameters is selected by comparing the RMSE corresponding to different hyperparameter values generated by training.