# Research on the Dynamic Characteristics Analysis and Power Control Method of Heat Pipe Reactors
## 4 Research on Power Control Method of HPR
### TD3 Algorithm  
The TD3 algorithm is a model-free, off-line, policy-based reinforcement learning method [34]. The TD3 algorithm is an improvement on the DDPG algorithm. And the DDPG algorithm originates from the DQN algorithm, combining the actor-critic network with the DQN algorithm. It not only solves the problems that the actor-critic network can only view one-sidedly and finds difficult to converge but also can handle the continuous control problem. However, the DQN algorithm is prone to overestimate the Q value, which also appears in the DDPG algorithm. To solve this problem, the DDQN algorithm eliminates overestimation through separating action selection and target Q value prediction. The TD3 algorithm uses three key techniques: the double network, target policy smoothing regularization, and the delayed policy update to further optimize the DDPG algorithm and avoid the overestimation problem. A double network refers to the use of two sets of critic networks, thus expanding the four neural networks of the DDPG algorithm into six neural networks of the TD3 algorithm. The network architecture of TD3 is shown in Figure 10. Through setting a dual critic network, the TD3 algorithm takes the minimum of output values \(Q_{tar\_1}\) and \(Q_{tar\_2}\) of two target networks as the prediction income, so as to calculate the temporal difference error, which makes the TD3 algorithm able to effectively deal with the overestimation problem. Target policy smoothing regularization is a regularization strategy based on the SARSA algorithm, which aims to solve the overestimated problem of value networks. The TD3 algorithm accomplishes the operation of target policy smoothing regularization through adding random noise to the output of the actor target network. The delayed policy update means that the update of the actor network lags behind the critic network. The actor network is updated through maximizing the cumulative expected reward, and it needs to use the critic network to evaluate the value of the action. If the critic network is unstable, the actor network will naturally oscillate as well. Therefore, the critic network needs to be updated more frequently than the actor network, i.e., it should wait  
Figure 10: Network architecture of TD3 algorithm. for the critic network to stabilize before updating the actor network. The update strategy adopted in the TD3 algorithm is to update the critic network once at each step, while the actor network is updated every \(d\) steps. The pseudo-code of the TD3 algorithm [35] is as follows. The algorithm first needs to create the actor and critic neural networks and randomly initialize the network parameters. At the beginning of the training, the algorithm undergoes a process of accumulating initial experiences, during which the algorithm will not be trained. The current critic networks are gradually updated in the direction of smaller TD error, while the current actor network of is updated in the direction of actions with higher reward. Compared with completely updating the target network in DQN, the method of updating TD3 algorithm is soft, that is, it merges two sets of network parameters using a small constant \(\tau\) (0 \(<\)\(\tau\)\(\ll\) 1). The soft update method of the TD3 algorithm results in a small change in the parameters of the target network and a relatively flat change in the calculated target value. This allows the algorithm to maintain some stability even if the target network is updated frequently. ```
Create Actor network \(\pi(s|\theta)\) and Critic network \(Q_{1}(s,a|\omega_{1})\) & \(Q_{2}(s,a|\omega_{2})\)  with randomly initializing the parameters \(\theta\), \(\omega_{1}\) and \(\omega_{2}\) respectively. Create target network \(\pi^{\prime}(s|\theta)\), \(Q^{\prime}_{1}(s,a|\omega^{\prime}_{1})\) and \(Q^{\prime}_{2}(s,a|\omega^{\prime}_{2})\), which set the parameters: \(\theta^{\prime}\leftarrow\theta,\omega^{\prime}_{i}\leftarrow\omega_{i}\), \(i=1,2\). Create an empty experience replay buffer \(R\).