# Research on the Dynamic Characteristics Analysis and Power Control Method of Heat Pipe Reactors
## 4 Research on Power Control Method of HPR
### TD3 Algorithm
In order to set the learning rate of the network reasonably, this paper adopts the exponential decay of learning rate strategy shown in Equation (14) to adjust the learning rate dynamically. The learning rate will decay at certain episodic intervals. However, the learning rate cannot be decayed infinitely, and the lower limit of learning rate decay should be set appropriately. \[Lr=Lr\cdot\mu\,\ \text{if}\ ep\ \text{mod}\ I=0 \tag{14}\]

where \(Lr\) is the learning rate, \(ep\) is the size of the current training curtain, and \(I\) is the episode interval between two adjacent learning rate decays. \begin{table}
\begin{tabular}{c c l} \hline \hline
**Status** & **Formula** & **Description** \\ \hline \(\mathbf{s_{0}}\) & \(n_{t}\) & Nuclear power at time \(t\) \\ \(\mathbf{s_{1}}\) & \(n_{ref}\) & Load power \\ \(\mathbf{s_{2}}\) & \(N_{t-1}\) & The speed of control drums at time \(t-1\) \\ \(\mathbf{s_{3}}\) & \(n_{ref}-n_{t}\) & The deviation between the actual nuclear power and the set value at \\  & & time \(t\) \\ \(\mathbf{s_{4}}\) & \(\frac{n_{t}-n_{t-1}}{h}\) & Rate of change in nuclear power \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Meaning of the 5 states. Reward is an extremely important part of reinforcement learning that allows the algorithm to evaluate what actions to take to get better results. In this paper, through designing a reasonable reward function, the reinforcement learning algorithm is provided with appropriate rewards and punishments, so that the algorithm learns a better control strategy and converges quickly. See Equation (15); the reward function consists of three parts, which are the load tracking part \(R_{t+1}^{TR}\), a control drums part \(R_{t+1}^{CD}\), and the nuclear power overrun part \(R_{t+1}^{OR}\). \[R_{t+1}=R_{t+1}^{TR}+R_{t+1}^{CD}+R_{t+1}^{OR} \tag{15}\]

Load tracking means that the controller controls the nuclear power to stabilize at the set power. The amount of the reward for load tracking is proportional to the absolute value of the deviation between the nuclear power and the load power, as in Equation (16). Where \(n_{t+1}\) is the actual nuclear power at the time \(t+1\), \(\varphi_{TR}\) is a positive number that represents the adjustable scaling factor of the load tracking reward. Since the algorithm is updated in the direction of increasing reward value, a negative sign is used in Equation (16) to ensure that the smaller the power deviation, the larger the reward \(R_{t+1}^{TR}\). \[R_{t+1}^{TR}=-\varphi_{TR}\cdot\left|n_{t+1}-n_{ref}\right| \tag{16}\]

The control drum is a kind of electric mechanical device with service life. If its speed changes frequently, it will accelerate the aging of the control drum and increase the possibility of its malfunction or even failure. Therefore, this paper introduces \(R_{t+1}^{CD}\) to optimize the rotation strategy of the control drums, as in Equation (17), where \(N_{t}^{CD}\) and \(N_{t-1}^{CD}\) are, respectively, the control drum's rotation speed at the time \(t\) and \(t-1\), and \(\varphi_{CD}\) is the positive scale factor of the control drum reward term. \[R_{t+1}^{CD}=-\varphi_{CD}\cdot\left|N_{t}^{CD}-N_{t-1}^{CD}\right| \tag{17}\]

In the process of regulating nuclear power, if the power of the nuclear power deviates too much from the set power, then this control is bound to fail. In order for the agent to

Figure 11: The structure of actor and critic networks. learn the strategy to deal with the above situation in time, the nuclear power overrun part \(R_{t+1}^{OR}\) in Equation (18) is used, where \(L\) is the penalty value given in case of nuclear power overrunning, \(n_{0}\) is the initial nuclear power, and \"20%FP\" refers to 20% of the full power. \[\left\{ \begin{array}{l}R_{t+1}^{OR}=-L,\text{ if }\left|n_{t+1}-n_{ref} \right|>\left|n_{0}-n_{ref}\right|+20\%FP\\ R_{t+1}^{OR}=0,\text{ otherwise}\end{array} \right.