# Reinforcement Learning for Autonomous Underwater Vehicles via Data-Informed Domain Randomization
## 5 Simulation and Experimental Results
### Sim-to-Real Tests  
The proposed data-informed domain randomization approach was applied to an underwater robot built on BlueRov2 from Blue Robotics, Inc. (Torrance, CA, USA), as shown in Figure 17a. It is a tethered ROV with six thrusters. With the given thruster configuration, the ROV is able to translate in three directions, roll, and yaw. The ROV is designed to have a sufficient restoring force, to keep itself horizontal. The ROV communicates with a laptop through the Robot Operating System (ROS) and receives thruster commands in the form of PWM signals. In order to provide real-time state estimation of the ROV, an underwater optical motion capture system from Nokov was implemented, which also communicates via ROS and publishes the pose topic in the form of three-dimensional vectors and quaternions. Since the field of view of cameras is often small in underwater applications, the motion capture system relies on 12 underwater cameras mounted on the walls of the tank. In addition, due to the short visibility distance, the reflective markers are of 30 mm, as shown in Figure 17b. The cameras emit blue lights (as shown in Figure 18a) and capture the pose of a rigid frame (as shown in Figure 18b) built on markers at 60 Hz. The motion capture system was calibrated with an L-shaped calibrator, which consists of four markers placed in the middle of the pool. The positioning system can reach a sub-centimeter resolution. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Scenario** & **Mean Error: \(\mathbf{u_{t}}\)** & **Mean Error: \(\mathbf{u_{t}}\)** \\ \hline “Swapping commands to & & \\ Thrusters 1 and 2” & 0.019 [m] & 0.41 [m] \\ “Manually reducing gain of & & \\ the thrusters” & 0.028 [m] & 0.39 [m] \\ “Setting negative gain to & & \\ Thruster 1” & 0.021 [m] & 1.16 [m] \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Mean errors from three scenarios. Figure 16: Mean square error of 100 trajectories of AUV regarding three scenarios. Figure 17: (**a**) BlueRov2 from Blue Robotivs, Inc.; (**b**) ROV with four reflective markers of 30 mm. The laptop receives the pose estimation from the motion capture system, implements the proposed RL approach, and sends the control signals to the ROV. The whole system is referred to as the testbed of the \"AUV\". In the future, a sonar-based localization approach and the proposed transfer RL algorithm will be implemented in the updated hardware of the ROV, making it an AUV. The dynamics model of the real AUV (i.e., the target dynamics model) is jointly determined by the body morphology and AUV velocities. The mass and other parameters are difficult to estimate. Note that only the dead zones and the saturation of the thrusters were simulated; however, the complex dynamics of the thrusters were not simulated in the source dynamics [26]. Therefore, there mismatches between the source dynamics model and the target model are inevitable. In addition, external disturbances were generated by a propeller fixed to the tank and an example of external disturbances is shown in Figure 19. Note that, when testing the algorithm, the AUV was detached from the force-torque sensor and the disturbance values are unknown. In one of the experimental tests, the AUV started from the position \(\mathbf{x}(0)=[0.18,-0.46,-0.4]^{T}\). Two trajectories were obtained: one \(\mathcal{T}_{s}\) obtained by \(\mathbf{u}_{s}\) and the second \(\mathcal{T}_{t}\) obtained by \(\mathbf{u}_{t}\). Both trajectories are illustrated in Figure 20a, where \(\mathcal{T}_{s}\) is shown by the red dashed line and \(\mathcal{T}_{t}\) by the blue solid line. The trajectories demonstrated that the transferred controller \(\mathbf{u}_{t}\) successfully regulated the AUV position, while the controller \(\mathbf{u}_{s}\) was unable to stabilize the AUV around the origin \(O_{t}\). The position regulation errors from both trajectories are shown in Figure 20b. The transferred controller \(\mathbf{u}_{t}\) stabilized the AUV in a sphere of radius 0.15 mm at the origin \(O_{t}\).