# A DDPG-Based USV Path-Planning Algorithm
## 2 Methodology
### Ddpg
The Actor-eval network emits real-time actions \(\mu(s_{t}|\theta^{\mu})\), which are enforced by the USV to engage with the environment;
2. The Critic-eval network produces \(Q\) values \(Q\big{(}s_{t},a_{t}|\theta^{Q}\big{)}\), utilized to assess the present state-action's worth;
3. The Actor-target network delivers \(\mu^{\prime}(s_{t+1}|\theta^{\mu\prime})\), calculating the destination \(Q\) value from the next optimal action \(a^{\prime}\) for state \(s_{t+1}\) in the observation replay reservoir;
4. The Critic-target network generates the objective \(Q\) value \(Q^{\prime}\Big{(}s_{t+1},\mu^{\prime}(s_{t+1}|\theta^{\mu\prime})|\theta^{Q^{ \prime}}\Big{)}\), and the UAV receives environmental rewards to derive the target \(Q\) value. The elementary notions linked to the DDPG algorithm encompass:  
1. **Deterministic Action Strategy \(\mu\)**: A function to calculate actions at each phase as \(a_{t}=\mu(s_{t})\). 2. **Policy Network**: This neural network emulates the function \(\mu\), termed a strategy network with parameters \(\theta^{\mu}\). 3. **Performance Metric \(J\)**: Also known as the performance objective, it gauges policy \(\mu\)'s efficacy. In an off-policy setting, it is described by \[J_{\beta_{(\mu)}}=\int_{S}\rho^{\beta}(S)\theta^{\mu}(S,\mu(S))dx=E_{s-p}\beta [Q^{\mu}(s,\mu(s))]\] (2) In this expression, \(S\) is the state created by the Agent-centric behavioral strategy; \(\rho^{\beta}\) is its distribution; \(Q^{\mu}(s,\mu(s))\) indicates the \(Q\) value resulting from choices under strategy \(\mu\). 4. **Strategy Gradient Definition**: This represents the gradient of performance objective function \(J\) relative to \(\theta^{\mu}\). 5. **Action-value Function (Q Function)**: In state \(s_{t}\), following action \(a_{t}\), and persistently applying policy \(\mu\), the anticipated value of \(R_{t}\) is \[Q^{\mu}(s_{t},a_{t})=E[r(s_{t},a_{t})+\gamma Q^{\mu}(s_{t+1},\mu(s_{t+1}))]\] (3) During the training phase of RL, the exploration of potentially superior strategies necessitates the introduction of stochastic noise into the action's decision-making protocol. This transformation alters the decision of action from a fixed pattern to a probabilistic one. Actions are then drawn from this probabilistic process and forwarded to the environment for implementation. This approach is referred to as the behavior policy, symbolized by \(\beta\), and categorized as off-policy. The UO (Uhlenbeck-Ornstein) random procedure, notable for its excellent correlation in temporal sequences, enables effective environmental exploration by the agent. Hence, in the context of this study, the UO stochastic process is employed for DDPG training. Concerning the training of the network, DDPG's focus lies in the continuous training and enhancement of the eval parameters within both Actor and Critic. Subsequently, after a specific duration, the target network's parameters are rejuvenated through the soft update algorithm. The updated formulation can be expressed as  
\[soft-update\left\{ \begin{aligned} \theta^{Q^{\prime}}& \Leftarrow\tau\theta^{Q}+(1-\tau)\theta^{Q^{\prime}}\\ \theta^{k}&\Leftarrow\tau\theta^{\mu}+(1-\tau)\theta^{ \mu}\end{aligned} \right. \tag{4}\]  
Here, \(\theta^{Q^{\prime}}\) and \(\theta^{\mu^{\prime}}\) correspond to the target network parameters, while \(\theta^{Q}\) and \(\theta^{\mu}\) are linked to eval network parameters. The parameter \(\tau\) is commonly set to 0.001. In the context of refining the parameters for the updated Critic network, the Critic network's loss is conceived as the MSE (mean square error), represented as  
\[L=\frac{1}{N}\sum_{i}\Bigl{(}y_{i}-Q\Bigl{(}S_{i},a\,z|\theta^{Q}\Bigr{)}\Bigr{)} ^{2} \tag{5}\]  
In this expression, \(y_{i}\). is delineated as  
\[y_{i}=R_{i}+\tau Q^{\prime}\Bigl{(}S_{i+1},\mu^{\prime}\,\Bigl{(}s_{i+1}|\theta ^{\mu^{\prime}}\Bigr{)}\Bigr{|}\theta^{Q^{\prime}}\Bigr{)} \tag{6}\]  
Here, \(\mu^{\prime}\) signifies the target network's parameters in the Critic, and \(Q^{\prime}\) represents the parameters of the target network in the Actor.