# Reinforcement Learning for Autonomous Underwater Vehicles via Data-Informed Domain Randomization
## 4 Transfer RL via Data-Informed Domain Randomization
### Data-Informed Domain Randomization  
Instead of directly adding Gaussian noise to the outputs of the simulated dynamics models, the noise is added to the parameters of dynamics models, allowing for more diverse distributions that can cover the actual distributions of the AUV states and dynamics. In each training episode, the parameters in the AUV dynamics model (3) include the inertia matrix \(\mathbf{M}\in\mathcal{M}\), the drag matrix \(\mathbf{D}\in\mathcal{D}\), the Coriolis matrix \(\mathbf{C}\in\mathcal{C}\), the vector \(\mathbf{g}\in\mathcal{G}\) of lumped gravity and buoyancy. The sets \(\mathcal{M}\), \(\mathcal{D}\), \(\mathcal{C}\), and \(\mathcal{G}\) of these parameters can be estimated [1]. The parameters are randomly sampled from \(\mathcal{M}\), \(\mathcal{D}\), \(\mathcal{C}\), and \(\mathcal{G}\). Besides, there are uncertainties regarding the transform matrix from thruster forces to the generalized force that acts through the AUV's center of mass. These uncertainties are also added to the Webots simulator. The control inputs \(\mathbf{tau}\) are subject to constraints from the phenomena of saturations and dead zones, which are determined by \(\underline{\tau}\) and \(\bar{\tau}\) and \(\mathbf{\tau}_{d}\), respectively. These constraints are also randomly chosen according to an estimated set. As illustrated in Figure 3, following the above-mentioned domain randomization approach, the dashed ellipse denotes the set \(\mathcal{F}_{s}\) of the AUV dynamics models that can be simulated in Webots. The solid ellipse presents the set \(\mathcal{F}_{t}\) of the real AUV under various conditions, such as thruster configurations and payloads. It is highly possible that the dashed ellipse and the solid ellipse only share a few common dynamics. Therefore, it is difficult to guarantee the stability of the AUV if the trained source controller \(\mathbf{u}_{s}\) is directly applied to the real AUV. \(\mathbf{u}_{s}\) might only be robust to the variations in the AUV dynamics in \(\mathcal{F}_{s}\cap\mathcal{F}_{t}\). However, this method is only applicable to the case where the set \(\mathcal{F}_{t}\) of target dynamics (the solid ellipse) is contained by the set \(\mathcal{F}_{t}\) of source dynamics (the dashed ellipse), as shown in Figure 4a. In this case, the trained controller \(\mathbf{u}_{s}\) is robust to the dynamics in the solid ellipse. The obtained controller \(\mathbf{u}_{s}\) maximizes the expectation of the objective function (5) over all possible dynamics in \(\mathcal{F}_{s}\). In other words, the expectation of the objective function (5) is not optimized over all possible dynamics in \(\mathcal{F}_{t}\). One way to reduce this gap is to adjust the parameter distribution so that \(\mathcal{F}_{s}\) is close to \(\mathcal{F}_{t}\). Many adaptive domain adaptation techniques have been studied, and they bias the distribution of parameters based on the trajectory data collected from the target domain \(\mathcal{D}_{s}\). The likelihood of the trajectory obtained from various simulated dynamics compared to an obtained trajectory from the real AUV can be computed and used to update the weights of the parameter particles. In each episode, particles with different weights are used to sample parameter values following the particle filter strategy, and to train the source controller \(\mathbf{u}_{s}\). This procedure identifies the system parameters and many other similar approaches can be explored in the future. However, the performance of the above strategy heavily relies on the assumption that \(\mathcal{F}_{t}\subset\mathcal{F}_{s}\). While treating unmodelled dynamics as noise, the nominal behavior of the actual AUV dynamics is contained by the set of source dynamics models. The actual system may exhibit dynamics that are not covered by \(\mathcal{F}_{s}\).