# Deep-Learning Model Selection and Parameter Estimation from a Wind Power Farm in Taiwan +
## 2 Relate Work
### Hyperparameter Optimization for Machine Learning Models  
The performance of deep learning models strongly depends on choosing a set of optimal hyperparameters. In other words, the prediction performance of the deep learning network model is influenced quite heavily by the choice of hyperparameters, hence it can incorporate a reinforcement learning optimisation process to improve the searching efficiency in model development. Hyperparameter tuning is an optimization problem where the objective function of optimization is unknown. Traditional optimization techniques such as the Newton method or gradient descent cannot be applied. Practically, data engineers find a set of hyperparameter values in the training phase to archive the best performance for machine learning models within a reasonable amount of time. Further, they continuously adjust hyperparameters for models with the selection of different sets of values for a learning algorithm and then compare the model performance to choose the best model. This process is called hyperparameter optimization. Figure 1: TCN-based architecture for wind power forecasting (dilated residual block \(k=2\)). Typically, there are mainly two kinds of hyperparameter optimization methods, i.e., manual search and automatic search methods. Manual search identifies the important parameters depending on the experience of expert users. The process of tuning hyperparameters is not easily reproducible. Moreover, as the number of hyperparameters and the range of values increase, it becomes quite difficult to manage [21]. To overcome the drawbacks of manual search, automatic search algorithms have been proposed, such as grid search [22; 23; 24; 25]. Mainly, grid search trains machine learning models with different values of hyperparameters in the training set and compares the performance according to evaluation metrics. Finally, grid search outputs hyperparameters that have the optimal performance. Two hyperparameter optimization approaches are compared in Table 1.  
Automatically tuning hyperparameters for the ideal model structure is a challenging task. Numerous automatic search techniques comprising machine learning approaches with optimised algorithms have been used for achieving high-precision performance. These AS schemes are summarised in Table 2. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Scheme** & **Features** & **Limitations** \\ \hline \multirow{4}{*}{Manual search (MS)} & The MS depends on the experience & \multirow{4}{*}{When the number of hyperparameters and the range of values increase, it becomes quite difficult to manage since humans are not good at handling high dimensional data.} \\  & of expert users to identify the & & \\  & important parameters. & & \\  & It is suitable for expert users to & & \\  & handle simple or low-dimensional & & \\  & hyperparameter tuning problems & & \\  & using the visualization tools. & & \\ \hline \multirow{4}{*}{Automatic search (AS)} & The AS trains machine learning & \multirow{4}{*}{Basically, the AS is a brute-force and exhaustive searching} \\  & models with selection of the & & \\  & optimal hyperparameters for a & & \\  & model that result in the most & & \\  & precise performance. & & \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Comparison of two hyperparameter optimization methods. Figure 2: Detailed diagram of the dilated residual block. Although the methods in Table 2 achieve automatic tuning approaches such as grid search and can theoretically obtain the near optimal value of the hyperparameters, they suffer from a high computational load with time cost.