# Reinforcement Learning for Autonomous Underwater Vehicles via Data-Informed Domain Randomization
## 5 Simulation and Experimental Results
### Sim-to-Sim Tests  
To test the proposed transfer RL via DDR, three scenarios with manually designed model mismatches were simulated. These scenarios were manually designed to reflect possible changes in the real AUV dynamics due to some sudden events. The first scenario involves changing the pose configurations of two thrusters to create a model mismatch between the source dynamics model \(\mathcal{F}_{s}\) and the target dynamics model \(\mathcal{F}_{t}\). As shown in Figures 9 and 10, the source controller \(\mathbf{u}_{s}\) was unable to stabilize the AUV with the target dynamics. This is because the changes in the pose configurations of thrusters with respect to the center of mass introduce a shift in the mapping, from the thrusts to the generalized. The mapping from \(\mathbf{\tau}_{t}\) to \(\mathbf{\tau}_{s}\) was quickly learned by a few episodes of data collection regarding the target dynamics. During these episodes, the performance of the position regulation in these episodes was poor and was not analyzed. After that, the mapping \(H\) was updated and the resultant \(\mathbf{u}_{t}\) was able to stabilize the AUV of the target dynamics, as shown in Figure 9. To illustrate the effectiveness of transfer RL, the trajectories of the AUV in the target domain under \(\mathbf{u}_{s}\) and \(\mathbf{u}_{t}\) are shown in Figure 10. The dashed line represents the trajectory obtained by \(\mathbf{u}_{s}\), and the solid line represents the one obtained by \(\mathbf{u}_{t}\). Let the error of position regulation be defined as the L2 norm of the AUV state. The errors of regulating AUV positions from \(\mathbf{u}_{s}\) and \(\mathbf{u}_{s}\) are shown in Figure 9. The disturbance effects and control forces are shown in Figure 11.  
A common issue is that the characteristics of a thruster may gradually change during its lifetime or vary suddenly due to certain events. In the second scenario, the gain and the maximum thrust of some thrusters were manually designed to simulate the case in which some thrusters were entangled by some seaweeds. As shown in Figure 12, when the source controller \(\mathbf{u}_{s}\) was tested on the target dynamics \(\mathcal{F}_{t}\), the reduced gain of some thrusters introduced additional unwanted torque along \(z\) axis, making the AUV oscillate its heading. The mapping from \(\mathbf{\tau}_{t}\) to \(\mathbf{\tau}_{s}\) was quickly learned over a few episodes. Again, the performance of the position regulation in these episodes was not analyzed. The trajectories of the AUV in the target domain under \(\mathbf{u}_{t}\) and \(\mathbf{u}_{t}\) are shown in Figure 12. The errors when regulating AUV positions from \(\mathbf{u}_{s}\) and \(\mathbf{u}_{s}\) are shown in Figure 13. \begin{table}
\begin{tabular}{c c c} \hline
**Index** & **Parameter** & **Value** \\ \hline
1 & \(m+X_{u}\) & 15.9 kg \\
2 & \(m+Y_{0}\) & 32.2 kg \\
3 & \(m+Z_{u0}\) & 14.5 kg \\
4 & \(I_{z}+N_{t}\) & 0.3 kg\(\cdot\)m \\
5 & \(X_{u}\) & \(-36.8\) kg \\
6 & \(X_{||u||u}\) & 374.7 kg\(\cdot\)s/m \\
7 & \(Y_{0}\) & \(-60.6\) \\
8 & \(Y_{||v|v}\) & 197.2 kg\(\cdot\)s/m \\
9 & \(Z_{w}\) & \(-112.2\) kg \\
10 & \(Z_{||u|v}\) & 153.2 kg\(\cdot\)s/m \\
11 & \(N_{r}\) & \(-0.32\) kg\(\cdot\)m \\
12 & \(N_{|r|r}\) & 0.95 kg\(\cdot\)m\(\cdot\)s \\ \hline \end{tabular}
\end{table} 
Table 1: Parameters of AUV models in numerical simualtions. Figure 11: Disturbance effects and control forces. Figure 10: Errors of the position regulation obtained by \(\mathbf{u}_{s}\) and \(\mathbf{u}_{t}\) for the AUV of \(\mathcal{F}_{t}\). Figure 9: Trajectories obtained by \(\mathbf{u}_{s}\) and \(\mathbf{u}_{t}\) for the AUV of \(\mathcal{F}_{t}\). The third scenario simulated an extreme case, where one of the four horizontal thrusters was damaged and a second thruster suddenly changed its phase in the PWM driver, causing it to rotate in the opposite direction. In this scenario, classical controllers such as PID or adaptive controllers may fail, since these methods often assume that the system reserves the positiveness of the gain matrix.