## 2 Materials and Methods
### The Reinforcement Learning Background
Entropy motivates the agent to sometimes pick actions that are not strictly reward-optimal in the short term, thereby aiding more extensive environmental exploration. The loss function for the actor in SAC is defined as  
\[\mathcal{L}_{\pi}\ =\ \mathbb{E}_{s\sim\mathcal{D}}\Big{[}\,\alpha\,\log\pi_{ \phi}(a\ |\ s)\ -\ Q_{\theta}(s,a)\Big{]}, \tag{17}\]  
where \(\alpha\) is the entropy coefficient governing the trade-off between exploration and exploitation. The critic in SAC estimates the action-value function \(Q_{\theta}(s,a)\) and is updated by minimizing the [PERSON] error:  
\[\mathcal{L}_{Q}\ =\ \mathbb{E}_{(s,a,r,s^{\prime})\,\sim\,\mathcal{D}}\Big{[} \Big{(}Q_{\theta}(s,a)\ -\ \Big{(}r\ +\ \gamma\,\mathbb{E}_{u^{\prime}\,\sim\,\pi_{\phi}}\big{[}Q_{\theta^{ \prime}}(s^{\prime},a^{\prime})\ -\ \alpha\,\log\pi_{\phi}(a^{\prime}\ |\ s^{\prime})\big{]}\Big{)}\Big{)}^{2}\Big{]}, \tag{18}\]  
where \(\theta^{\prime}\) are the parameters of the target network (a copy of the critic network). In Twin Delayed Deep Deterministic Policy Gradient (TD3), the policy is deterministic and approximated by the actor network \(\mu_{\phi}(s)\). To enhance the training stability and reduce value overestimation, TD3 introduces two critics and delays updates to the actor. The critic loss function is given by  
\[\mathcal{L}_{Q}\ =\ \mathbb{E}_{(s,a,r,s^{\prime})\,\sim\,\mathcal{D}}\Big{[} \big{(}Q_{\theta_{i}}(s,a)-y\big{)}^{2}\Big{]}, \tag{19}\]  
where the target value \(y\) is computed as  
\[y\ =\ r\ +\ \gamma\,\min_{i=1,2}Q_{\theta^{\prime}_{i}}(s^{\prime},\,\mu_{ \phi^{\prime}}(s^{\prime})\,+\,\epsilon), \tag{20}\]  
and \(\epsilon\) represents the noise added to the action for exploration. The parameters \(\theta^{\prime}_{i}\) and \(\phi^{\prime}\) refer to the target networks. The actor in TD3 is updated by maximizing the critic's predicted value:  
\[\mathcal{L}_{\pi}\ =\ -\mathbb{E}_{s\,\sim\,\mathcal{D}}\Big{[}Q_{\theta_{1}} \big{(}s,\,\mu_{\phi}(s)\big{)}\Big{]}. \tag{21}\]  
Truncated Quantile Critics (TQCs) extends TD3 by estimating the action-value function via quantiles, enabling a more robust treatment of uncertainty. The critic in TQC approximates the distribution of \(Q(s,a)\) and minimizes the quantile Huber loss:  
\[V^{\pi}(s)=\mathbb{E}_{\pi}\big{[}G_{t}\bigm{|}S_{t}=s\big{]}. \tag{13}\]0where \(\delta_{ij}=r+\gamma\ \hat{Z}_{\theta_{j}}(s^{\prime},a^{\prime})\ -\ \hat{Z}_{\theta_{i}}(s,a)\), \(\hat{Z}_{\theta}(s,a)\) are the estimated quantiles, \(N\) is the total number of quantiles, and \(\rho_{\tau_{l}}\) is the Huber loss function quantile. The actor in TQC is updated in a manner similar to that in TD3 but uses the mean of the lowest quantiles to compute the action-value:  
\[V^{\pi}(s)=\mathbb{E}_{\pi}\big{[}G_{t}\bigm{|}S_{t}=s\big{]}. \tag{13}\]1  
where \(M\) is the number of the lowest quantiles selected. In these algorithms, the policy can be deterministic (TD3, TQC) or stochastic (SAC). A deterministic policy maps states directly to actions, whereas a stochastic policy defines a probability distribution over the possible actions in each state, thereby promoting exploration of the environment. The actor and critic network parameters are updated via stochastic gradient methods (e.g., the Adam optimizer), using the agent's experience, stored in a replay buffer \(\mathcal{D}\). By minimizing the relevant loss functions on these collected samples, the agent learns the optimal control policies in continuous and stochastic environments, which is critical for path-planning applications in AUVs.