# Research on the Dynamic Characteristics Analysis and Power Control Method of Heat Pipe Reactors
## 4 Research on Power Control Method of HPR
### TD3 Algorithm
\tag{18}\]

In summary, in order to make the TD3 algorithm applicable to HPR power control, the hyperparameters of the algorithm are set in this paper, and most of them are listed in Table 3.

## 5 Results and Discussion

The training condition in this paper is limited to between 100%FP and 10%FP. First, it decreases from 100%FP with 10%FP as a power step down to 10%FP, and then with the same power step up to 100%FP. Each episode running time is 180 s. During training, the fourth-order Runge-Kutta method (RK4) is used as the solution algorithm of the HPR dynamic model, and the calculation step is 0.001 s. The sampling time of the RL controller is 0.1 s; that is, the controller controls every 0.1 s. In order to verify the effectiveness of the RL controller, three simulations are designed to analyze its control performance. ### The Optimization Results of the RL Controller

During a training session, the curves of episode reward, average reward, and Q value of each episode changes along with the training and are shown in Figure 12. It can be seen that the nuclear power limit is often exceeded during the early training of the agent, which leads to the training in one episode being terminated in advance, so that the reward obtained by the RL agent is tiny. After a period of training, the agent gains enough operational experience to gradually learn the control strategy of nuclear power. At the end of training, the upward trend of episode reward, average reward, and Q value gradually stops; that is, the training of the RL agent has converged. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Hyper-Parameters** & **Values** & **Hyper-Parameters** & **Values** \\ \hline Sample time \(T_{s}\) & 0.1 s & Gradient threshold & 1 \\ Step size of RK4 \(h\) & 0.001 s & Optimization algorithm & Adam \\ Time simulated per episode \(T_{max}\) & 180 s & Frequency of & 10 steps \\ Number of episodes \(Ep\) & 600 & Soft update ratio \(\tau\) & \(5\times 10^{-3}\) \\ Size of experience buffer \(R\) & \(10^{6}\) & Action space & (\(-1\), 1) \\ Size of mini batch \(N\) & 1024 & The initial noise variance & 0.1 \\ Discount rate \(\gamma\) & 0.99 & Decay rate of noise variance & \(10^{-4}\) \\ Initial learning rate of Actor network \(\alpha_{1}\) & \(10^{-3}\) & The minimum noise variance & 0.001 \\ Interval of learning rate decay \(l\) & 2 episodes & \(\varphi_{TR}\) & 10 \\ Initial learning rate of Critic network \(\alpha_{2}\) & \(10^{-4}\) & \(\varphi_{CD}\) & 0.1 \\ Learning rate & 0.98 & \(L\) & 100 \\ decay rate \(\mu\) & & & \\ Minimum & \(10^{-5}\) & & \\ learning rate & & & \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Hyper-parameters of TD3 algorithm. The interference of the HPR system in operation is often varied and complicated, and the nuclear power changes frequently. To verify the control performance of the controller, a complex condition with multiple disturbances and variable load power is designed. The following disturbance or variable power instructions are introduced to the reactor system:

1. Linear introduction of \(-10\) PCM reactivity into the reactor between 40 s and 50 s.
2. The load power decreases linearly by 10%FP from 100 s to 130 s.
3. The load power is reduced to 70%FP at 200 s.
4. The load power is restored to 100%FP at 300 s.

Among them, the traditional PID control method is used as the control method in this paper, and the comparison between PID's control effect and RL's control effect is shown in Figure 14. The parameters of the PID controller were obtained automatically using a PID tuner in Simulink under the 100 FP-90%FP condition, and the PID parameters were \(K_{P}=1.02\), \(K_{I}=0.01\), and \(K_{D}=3.21\). Table 5 states the control performance of the two control methods. It is worth noting that the dynamic characteristics in Table 5 refer to the maximum value of characteristics in the whole simulation process. In the simulation, RL control has a faster response speed and can track load changes more quickly than the PID control.