# Reinforcement Learning for Autonomous Underwater Vehicles via Data-Informed Domain Randomization
## 2 Problem Formulation
The control inputs \(\mathbf{\tau}\) are subject to constraints from the phenomena of saturations and dead zones, which are, respectively, given by  
\[\underline{\mathbf{\tau}}\leq\mathbf{\tau}\leq\mathbf{\tau}\] \[|\mathbf{\tau}|\geq\mathbf{\tau}_{d}, \tag{2}\]  
where \(\underline{\mathbf{\tau}}\) and \(\mathbf{\tau}\) denote the saturation bounds of the control inputs, \(|\cdot|\) denotes the absolute operation, and \(\mathbf{\tau}_{d}\) denotes the dead band. Let \(\mathcal{U}\) denote the set of control that satisfies Equation (2). However, the feasible set \(\mathcal{U}\) may not be accurately estimated due to varying power supply and current conditions. **Problem 1** (Position Regularization Problem). : _Design a controller that can bring the \(\Delta\)UV starting at \(\mathbf{\eta}_{i}(0)\) to the origin \(\mathbf{O}_{i}\), subject unknown dynamics model (1), and the constraints of the dead zones and saturations (2)._  
This paper explores the feasibility of RL techniques to train a controller from numerical simulations built on Equation (1). The continuous-time model in (1) is converted into a discrete-time model by [PERSON]'s first-order expansion, as shown below:  
\[\mathbf{\eta}_{i}\triangleq[x_{i},y_{i},z_{i},\theta_{i}]^{T},\]0  
where  
\[\mathbf{\eta}_{i}\triangleq[x_{i},y_{i},z_{i},\theta_{i}]^{T},\]1  
and \(\Delta t\) is the sampling time. The forward dynamics model in (3) is denoted as \(\mathcal{F}_{s}\). This model can be simulated with estimates of unknown parameters by the Webots simulator and the controller can be trained to solve Problem 1 of the simulated system (3). The controller \(\mathbf{u}_{s}\) outputs the control vector \(\mathbf{\tau}(t)\) at time \(t\), i.e.,  
\[\mathbf{\eta}_{i}\triangleq[x_{i},y_{i},z_{i},\theta_{i}]^{T},\]2  
where \(\mathbf{e}^{T}(t)\) is the error of position regulation viewed in the body frame and is obtained by  
\[\mathbf{\eta}_{i}\triangleq[x_{i},y_{i},z_{i},\theta_{i}]^{T},\]3  
The transformation matrix \(\mathbf{T}(\mathbf{\eta}_{i})\) transforms a point in the body frame to the earth-fixed frame. The controller \(\mathbf{u}(\cdot)\) maps from the AUV velocity and position errors to control inputs. Let \(\mathbf{y}(t)\triangleq[\mathbf{\
u}^{T}(t),\mathbf{e}^{T}(t)]^{T}\) and refer \(\mathbf{x}(t)\) as the AUV state at time \(t\). It is assumed that the AUV state can be acquired with small noises at high frequencies. The controller trained in the source domain \(\mathcal{D}_{s}\) under the source dynamics models \(\mathcal{F}_{s}\) (see Equation (3)) is denoted as \(\mathbf{u}_{s}\), referred to as the source controller. The source controller would be applied to a target dynamical system \(\mathcal{F}_{t}\) in a target domain \(\mathcal{D}_{t}\). The forward dynamics model \(\mathcal{F}_{t}\) presents a real AUV system or a simulated AUV with a different dynamics model. Since the direct application of \(\mathbf{u}_{s}\) on \(\mathcal{F}_{t}\) brings unsatisfactory results, \(\mathbf{u}_{s}\) has to be transferred to obtain \(\mathbf{u}_{t}\) for better performance. **Problem 2** (Controller Transfer Problem). : _Design an approach to learn a source controller \(\mathbf{u}_{s}\) for the source dynamics \(\mathcal{F}_{s}\) and to transfer \(\mathbf{u}_{s}\) to the target dynamical system \(\mathcal{F}_{t}\) and obtain a target controller \(\mathbf{u}_{t}\), such that the Problem 1 of the target system \(\mathcal{F}_{t}\) can be solved by \(\mathbf{u}_{t}\)._  
Model-free RL can deal with uncertainties and train controllers on deterministic dynamical systems with various parameters to improve the stability of the trained controller \(\mathbf{u}_{s}\) the. However, dynamics models from through-parameter randomization may not cover those in the target domain. To this end, in this paper, a data-informed domain randomization approach is developed to solve Problem 2. In the meantime, the model mismatches between the source and target dynamics models have to be quickly estimated online to efficiently adjust the controller \(\mathbf{u}_{t}\).