## 2 BIT* Algorithm Procedure
### Improved BIT* + Energy Consumption Equation + TD3 Hybrid Path Planning Strategy Model
#### 3.3.8 Parameter Configuration for the TD3 Deep Reinforcement Learning Strategy Model  
To expedite the implementation of the decision-making model, we opted for the PyTorch 2.5.0 deep learning framework and the Python programming language to train the TD3 model. The network architecture of the TD3 model examined in this research primarily comprises one Actor network and two Critic networks (twin Q-networks). The detailed structure can be referenced in the definitions of the Actor and Critic classes within the code. Figure 3 illustrates the detailed workflow of the TD3 algorithm. As shown in the figure, the TD3 algorithm adopts an Actor-Critic architecture, where the Actor network is responsible for outputting actions based on the current state and the Critic network is responsible for evaluating the value of actions. The TD3 algorithm also introduces target networks and an experience replay mechanism to improve the stability and efficiency of training. The parameters of the target networks are copied from the main networks through a soft update method, and the experience replay mechanism stores the historical data of the agent's interaction with the environment for subsequent training. The TD3 algorithm updates the parameters of the Critic network by minimizing the temporal difference (TD) error and updates the parameters of the Actor network through the policy gradient method. In this research, the core objective of the TD3 (Twin Delayed Deep Deterministic Policy Gradient) algorithm is to generate a path that is smoother, shorter, and more conducive to Unmanned Surface Vehicle (USV) execution by incorporating turning angle cost, speed decay mechanisms, and path length rewards. Specifically, given the initial path \(P=p1,p2,\ldots,pn\) produced by bit*, two points \(pi\) and \(pj(i<j)\) are selected along the trajectory. To mitigate the potential for abrupt changes in heading at the junctures between the optimized segment and the original path, their immediate predecessor \(pi-1\) and successor \(pi+1\) are also considered. The optimization procedure is delineated as follows:  
1. Turning Angle Cost: The turning angle cost penalizes sharp turns in the trajectory, promoting smoothness. For each point \(p_{k}(i\leq k\leq j)\) on the path segment, the turning angle \((\theta_{k})\) is computed using the vectors \(\left(\overset{\rightarrow}{v}_{\text{prev}}=p_{k}-p_{k-1}\right)\) and \(\left(\overset{\rightarrow}{v}_{\text{next}}=p_{k+1}-p_{k}\right)\). \[\theta_{k}=\arccos\left(\frac{\overset{\rightarrow}{v}_{\text{prev}}\cdot \overset{\rightarrow}{v}_{\text{next}}}{\left|\overset{\rightarrow}{v}_{ \text{prev}}\right|\cdot\left|\overset{\rightarrow}{v}_{\text{next}}\right|} \right). \tag{10}\]  
The turning angle cost, denoted as \(\left(C_{\text{angle}}\right)\), is defined as the summation of all turning angles along the segment. \[C_{\text{angle}}=\sum_{k=i}^{j}\theta_{k}. \tag{11}\]  
2. Velocity Decay Mechanism: To mitigate the impact of turning angles on velocity, a velocity decay mechanism is introduced. For each point \((p_{k})\), its velocity \((v_{k})\) is adjusted based on the turning angle \((\theta_{k})\). \[v_{k}=v\cdot\cos(\theta_{k}), \tag{12}\]  
where \(V\) represents the nominal velocity of the USV. The velocity decay mechanism ensures that the robot automatically decelerates during turns, thereby enhancing the smoothness and safety of the trajectory. 3. Path Length Reward: The path length reward incentivizes the generation of shorter paths. For the optimized path segment \(\left(P^{\prime}=\left\{p^{\prime}_{i},p^{\prime}_{i+1},\ldots,p^{\prime}_{j} \right\}\right)\), its path length \((L(P^{\prime}))\) is computed using the Euclidean distance. \[L\left(P^{\prime}\right)=\sum_{k=i}^{j-1}|p^{\prime}_{k+1}-p^{\prime}_{k}|. \tag{13}\]  
Figure 3: Flowchart of the TD3 algorithm. The path length reward, denoted as \(\left(R_{\text{length}}\right)\), is defined as the negative of the path length. \[R_{\text{length}}=-L\big{(}P^{\prime}\big{)}. \tag{14}\]  
4.