# Reinforcement Learning for Autonomous Underwater Vehicles via Data-Informed Domain Randomization
## 2 Problem Formulation  
This paper studies the pose regulation problem of the AUV subject to model uncertainties and other input constraints, such as dead zones and input saturations. Let \(X_{i}Y_{i}Z_{i}\) denote the earth-fixed reference frame and \(X_{b}Y_{b}Z_{b}\) denote the body-fixed reference frame [1], as shown in Figure 1. Axis \(O_{i}X_{i}\) and Axis \(O_{i}Y_{i}\) are in the horizontal plane, Axis \(O_{i}Z_{i}\) is the gravity direction. The body-fixed frame \(X_{b}Y_{b}Z_{b}\) is attached to the AUV center and the \(O_{b}X_{b}\) axis coincides with the AUV heading. The \(O_{b}Y_{b}\) axis is along the starboard direction. The AUV pose in the frame \(Z_{i}Y_{i}Z_{i}\) is defined as its position \([x_{i},y_{i},z_{i}]^{T}\) and attitude \([\rho,\psi,\theta]^{T}\). Similar to many AUVs, the restoring force is often sufficiently large to maintain its roll \(\rho\) and pitch \(\psi\) close to zero under all circumstances. The restoring force can be manually designed by adjusting the distance between the mass center and the buoyancy center of the AUV. As a result, in this paper, the studied AUV is described by motions of four Degrees Of Freedom (DOFs), including the linear motions along the \(O_{b}X_{b}\), \(O_{b}Y_{b}\), and \(O_{b}Z_{b}\) directions and the yaw motion around the \(O_{b}Z_{b}\) axis. Then, the AUV state pose \(\mathbf{\eta}\) with respect to the earth-fixed frame \(Z_{i}Y_{i}Z_{i}\) is redefined as  
\[\mathbf{\eta}_{i}\triangleq[x_{i},y_{i},z_{i},\theta_{i}]^{T},\]  
where \(x_{i}\), \(y_{i}\), and \(z_{i}\) are the AUV's position, and \(\theta_{i}\) denotes the AUV's heading in the earth-fixed frame. Then, the AUV's velocity \(\mathbf{v}_{i}\) in \(X_{i}Y_{i}Z_{i}\) is defined as  
\[\mathbf{v}_{i}\triangleq\dot{\mathbf{\eta}}_{i}=[u_{i},v_{i},w_{i},\omega_{i}]^{T},\]  
where \(u_{i}\), \(v_{i}\), and \(w_{i}\) are translational velocities in the x-, y-, and z-axes in the inertial frame, and \(\omega_{i}\) is the angular velocity along the z-axis in the inertial frame. Then, the generalized velocity in \(X_{b}Y_{b}Z_{b}\) is given as  
\[\mathbf{v}=[u_{b},v_{b},w_{b},\omega_{b}]^{T},\]  
where  
\[[u_{b},v_{b},w_{b}]^{T}=\mathbf{R}(\theta_{i})[u_{i},v_{i},w_{i}]^{T},\]  
\[\omega_{b}=\omega_{i},\]  
and \(\mathbf{R}(\theta+i)\) denotes the transform from Frame \(X_{i}Y_{i}Z_{i}\) to Frame \(X_{b}Y_{b}Z_{b}\). Let the generalized control \(\mathbf{\tau}\) from lumped thrusts in Frame \(X_{b}Y_{b}Z_{b}\) be denoted as \(\mathbf{\tau}\triangleq[\tau_{x},\tau_{y},\tau_{z},T_{z}]^{T}\), which may not act through the center of mass. In fact, the mapping between thrusts to the generalized control \(\mathbf{\tau}\) may not be accurately known, due to manufacturing issues. The dynamics model of the AUV is often described in the body frame [22] as  
\[\mathbf{M}\dot{\mathbf{v}}+\mathbf{C}(\mathbf{v})\mathbf{v}+\mathbf{D}(\mathbf{v})\mathbf{v}+\mathbf{g}(\mathbf{\eta}) =\mathbf{\tau}, \tag{1}\]  
where \(\mathbf{M}\) denotes the inertia matrix with added mass from motion in the water, \(\mathbf{D}\) denotes the coefficient matrix of the drag forces, \(\mathbf{C}\) denotes the Coriolis matrix, and \(\mathbf{g}\) is the lumped vector of gravity and buoyancy. The value of \(\mathbf{g}\) can be assumed to be constant, while the values of other matrices are partially determined by the velocities of the AUV and water  
Figure 1: Coordinate frames and control inputs of the AUV. current, and are, therefore, difficult to estimate. The dynamics model in Equation (1) only captures the main aspects of the actual system under mild conditions, leaving some effects unmodeled, such as thruster dynamics.