# Research on the Dynamic Characteristics Analysis and Power Control Method of Heat Pipe Reactors
## 4 Research on Power Control Method of HPR
### TD3 Algorithm
Initialize sampling time \(T_{s}\), maximum simulation time \(T_{max}\), number of episodes \(Ep\), and calculate the maximum steps \(St=T_{max}/T_{s}\). forepisode = 1 to \(Ep\)do  Run the simulation environment and observe the initial state \(s_{1}\) of the environment forstep = 1 to \(St\)do  Select action based on policy network and noise: \(a_{t}=\pi(s_{t}|\theta)+\epsilon\), \(\epsilon=N(0,\sigma^{2})\);  Execute \(a_{t}\), then calculate the reward \(r_{t}\) and observe the state \(s_{t+1}\)  Store experience \((s_{t},a_{t},r_{t},s_{t+1})\) in replay buffer \(R\). if Number of lines in replay buffer > minibatch:  Randomly sample a minibatch of experience from \(R\): \(\{s_{k},a_{k},r_{k},s_{k+1}\}_{k=1}^{N}\); \(a_{k+1}=\pi(s_{k+1})+\epsilon\), \(\epsilon=clip(N(0,\sigma^{2}),-\epsilon,c)\); \(\gamma=r_{k}+\gamma\cdot min\{Q^{\prime}_{i}(s_{k+1},a_{k+1})\}_{i=1,2}\)  Update the parameters of the Critic network:



### Settings of TD3 Algorithm

When using the TD3 algorithm to create the HPR controller, it is necessary to focus on the states \(\mathbf{s}\), the structure of the actor-critic network, the reward functions, the decay strategy of the learning rate, and other hyperparameters. During the training and use of an RL agent, it needs to continuously acquire the states of the environment to determine the next action. There is a limited number of states acquired by the intelligent body. On the one hand, too many states will lead to useful information being overwhelmed; on the other hand, the more states, the larger the size of the neural network which has more parameters need to be trained, thus leading to an increase in training time. Therefore, in this paper, based on practical needs and experience, 5 states are selected as a set of states \(\mathbf{s}=[\mathbf{s_{0}},\mathbf{s_{1}},\mathbf{s_{2}},\mathbf{s_{3}},\mathbf{s_{4}}]\). The meaning of each state is shown in Table 2. \(\mathbf{s_{0}},\mathbf{s_{1}}\) are the basic states of the controlled parameters, which represent the actual and set values of nuclear power, respectively. \(\mathbf{s_{2}}\) is the speed of control drums at the previous moment. The speed of control drums is both the output and the input of the agent, which is beneficial for the agent to learn the advantages and disadvantages of the control action to produce a better control strategy. \(\mathbf{s_{3}}\) is the difference between the actual nuclear power and the load power, indicating the deviation in the nuclear power. And \(\mathbf{s_{4}}\) indicates the rate of change in nuclear power and represents how fast the power changes. Both the actor network and the critic network adopt the double network structure form of the DQN algorithm; that is, each network exists in both the current network and target network, and the current network has the same network structure as the target network, but the network parameters are different. Therefore, the TD3 algorithm contains two types of neural networks and a total of six neural networks. The structures of the actor network and the critic network are shown in Figure 11. The actor network uses a three-layer fully connected neural network, and the activation function uses the ReLU function and the tanh function, where the tanh function is used to restrict the output of the actor network between \(-1\) and \(1\). The critic network has two branches, one for receiving the observed states and the other for receiving the control drums' speed. The two branches are combined by ADD layers and processed by a three-layer fully connected network to obtain the evaluation values of states and actions. As for the learning rates of actor networks and critic networks, they cannot be set too large to prevent the network from hovering around the optimal value and not converging. At the same time, too small a learning rate will lead to slow convergence of the network and even fall into a local optimum.