# Reinforcement Learning for Autonomous Underwater Vehicles via Data-Informed Domain Randomization
## 3 Reinforcement Learning  
This section briefly describes the RL approach to solve Problem 1 of the simulated AUV. As shown in Figure 2, the learning procedure interacts with the simulator and collects the trajectories and rewards. The task in Problem 1 is converted to an optimization problem as  
\[\max J=\sum_{t=1}^{\infty}\gamma^{t}r(t), \tag{5}\]  
where \(\gamma\) is a discount factor that penalizes long-term rewards and the reward function \(R\) is defined as follows. \[r(\mathbf{e},\mathbf{v},\mathbf{\tau})\triangleq-\alpha_{1}{}^{T}\|\mathbf{e}\|-\alpha_{2}{}^ {T}\|\mathbf{v}\|-\alpha_{3}{}^{T}\|\mathbf{\tau}\|, \tag{6}\]  
where \(\|\cdot\|\) denotes \(L2\)-norm. The term \(\mathbf{e}\in\mathbb{R}^{3}\) is the regulation error on positions, \(\mathbf{v}\in\mathbb{R}^{4}\) is the AUV's generalized velocity and it has to be zero when the AUV is at the origin in Frame \(X_{i}Y_{i}Z_{i}\), and \(\mathbf{\tau}\in R^{4}\) denotes the control inputs. The weights \(\mathbf{\alpha}_{1}\), \(\mathbf{\alpha}_{2}\), and \(\mathbf{\alpha}_{3}\) are positive parameters and are defined by users. In this paper, they were chosen such that \(\mathbf{\alpha}_{1}\gg\mathbf{\alpha}_{2}\gg\mathbf{\alpha}_{3}>0\). The reward function has to be designed or learned to reflect the control perpurse, which is a hot topic as it affects the convergence process in the learning and stability performance of the AUV [23]. The optimization of the objective (5) is solved against the equality constraints from the AUV dynamics model (3) and the inequality constraints from the control dead zones and saturations (2), outputting the source controller \(\mathbf{u}_{s}\). These two types of constraints are simulated in the Webots simulator and are rediscovered by the interaction between RL and the simulator by the data \(\{\mathbf{x}_{s}(t),\mathbf{\tau}_{s}(t),\mathbf{x}_{s}(t+1),r_{s}(t)\}_{t\in\mathbb{I}}\), where \(I_{t}\) is the index set. RL is essentially a exploration-and-exploitation algorithm that updates its controller through interactions between the learning agent and the simulator (i.e., \(\{\mathbf{x}_{s}(t),\mathbf{\tau}_{s}(t),\mathbf{x}_{s}(t+1),r_{s}(t)\}_{t\in\mathbb{I}}\)). After being fitted in Markovian decision processes, the objective function is the cumulated rewards and should be maximized. The reward function is defined in Equation (5) [24]. In the \(t\)th iteration, the control inputs \(\mathbf{\tau}(t)\in\mathcal{U}\) are chosen by the controller \(\mathbf{u}_{s}\) based on the state \(\mathbf{x}_{s}(t)\) (i.e., the current regulation error \(\mathbf{e}_{t}\) and velocities \(\mathbf{mu}_{t}\)). The simulator receives the control inputs \(\mathbf{\tau}(t)\) and outputs the AUV state \(\mathbf{x}_{s}(t)\) after a one-step simulation and a reward value defined in Equation (6). The obtained data \(\{\mathbf{x}_{s}(t),\mathbf{\tau}_{s}(t),\mathbf{x}_{s}(t+1),r_{s}(t)\}_{t\in\mathbb{I}}\) are used to update the controller network \(\mathbf{u}_{s}\) and the critic network \(V_{s}\), as shown in Figure 2. The critic \(V_{s}\) is essentially the objective function evaluated at the state \(\mathbf{x}_{s}\), conditioned on the controller \(\mathbf{u}_{s}\), which is not optimal before the convergence of the learning process. The controller \(\mathbf{u}_{s}\) is then updated by maximizing the critic \(V_{s}\) at given \(\mathbf{x}_{s}\). The procedures are briefly outlined in Algorithm 1. Figure 2: Learning mechanism of reinforcement learning. While there are many methods to train the controller network, this paper adopts the Soft Actor-Critic (SAC) [25], which is an open-source off-policy learning algorithm. As reported by many researchers, SAC can achieve sufficient results in benchmark problems.