# Deep-Learning Model Selection and Parameter Estimation from a Wind Power Farm in Taiwan +
## 3 Determining the Hyperparameters for the Developed Model with Q-Learning Scheme
### Hyperparameter Selection for the TCN Model Using Q-Learning Algorithm  
Inspired by [PERSON], [PERSON], and [PERSON] in [27], we adapted Q-learning agents to split the design space into independent smaller design sub-spaces such that the agents fine-tune the hyperparameters for the assigned layer and provide faster, yet accurate design space search. Thus, the optimal parameters for the developed wind power prediction model with Q-learning algorithm are analyzed as follows. In the training phase, the present study used a TCN model pre-trained on the experimental dataset containing collected wind turbine data from the Chang-Bin wind power plant in Taiwan from 2020 to 2021 (2-year period) and incorporated a Q-learning algorithm to determine the optimal parameters for the developed models in the training process (Figure 3). As shown in Figure 4, the selection process for architecture parameters is modelled as an MDP. To solve the problem of exploring the proper setting of hyperparameters for the TCN model, this study proposed a prediction model involving Q-learning with the reinforcement learning approach as follows. Initially, the loss function of the Q-learning model, \(\mathrm{L}(\theta)\), is given by  
\[\mathrm{L}(\theta)=E\,\left[\left(r_{k}(s,a;\theta)+\gamma\,\max_{a}\!\! \mathrm{Q}_{k}\big{(}s^{\prime},a^{\prime};\theta\big{)}-\,\mathrm{Q}_{k}(s,a; \theta)\big{)}\right], \tag{13}\]  
where \(\,r_{k}(s,a;\theta)\) represents the reward value of the current state \(s\), \(\theta\) is the parameter from the previous iteration, \(s^{\prime}\) is the state after performing action \(a\), and \(\gamma\) is a number between 0 and 1, \(0\leq\gamma\leq 1\), called the discount factor, which trades off the importance of sooner against later rewards. Further, \(\gamma\) may also be interpreted as the likelihood of succeeding at every step \(k\). \(\mathrm{Q}(s,\ a\ ;\theta)\) denotes the quantity of a state-action pair (s, \(a\)). By performing action \(a\), the agent can move from a state \(s\) to a new state \(s^{\prime}\), in which each state provides the agent with a reward. As described above, the agent is used for maximising its total reward by learning the optimal action for each state. Figure 4: Update process for \(Q\)-value computation. Figure 3: Architecture of a TCN model for wind power forecasting enhanced with Q-learning algorithm. To minimise the errors between the predicted values and actual outputs (i.e., maximise the reward to minimise the prediction errors between prediction value and real output) for wind power forecasting, we define the reward (\(R\)) between state \(s\) and new state \(s^{\prime}\) as  
\[R=\Big{(}TCN_{Loss(s)}-TCN_{Loss(s^{\prime})}\Big{)}. \tag{14}\]  
In finite state-action spaces, the Q-learning approach can solve for \(Q\)-value using an update rule as follows (Figure 4). \[Q_{k+1}(s,a;\theta)=Q_{k}(s,a;\theta)+\alpha[r_{k}(s,a;\theta)+\gamma_{a}\;Q_ {k}(s^{\prime},a^{\prime};\theta)-Q_{k}(s,a;\theta)], \tag{15}\]  
where \(Q_{k+1}(s,a;\theta)\) is a new \(Q\)-value of the state-action pair for state \(s\), \(Q_{k}(s,a;\theta)\) is the \(Q\)-value of the state-action pair for current state \(s\); \(r_{k}(s,a;\theta)\) is the reward value of the state-action pair for current state \(s\), and \(Q_{k}(s^{\prime},a^{\prime};\theta)\) represents the \(Q\)-value of the state-action pair for a new state \(s^{\prime}\). In summary, the \(Q\)-learning computation was performed in the following five sub-steps [21,22]:  
1. Initialise \(Q\)-values \(Q(s,\;a;\theta)\) arbitrarily for all state-action pairs. 2. For life, or until learning is stopped. 3. The agent selects action (\(a\)) in the current state (\(s\)) based on the current \(Q\)-value and estimates state-action pair \(Q(s,a;\theta)\) using Equation (15). 4. Applying Equation (15), \(Q\)-values are updated after a new state and reward are observed. 5. Calculate L(\(\theta\)), and then stop updating when \(\Delta\)L(\(\theta\)) \(<\epsilon\).