# Reinforcement Learning for Autonomous Underwater Vehicles via Data-Informed Domain Randomization
## 4 Transfer RL via Data-Informed Domain Randomization
### Controller Transfer across Domains  
Finding correspondences between the dynamics and controller across domains is key to improving the learning efficiency in the target domain [13]. In Problem 2, the position regulation tasks of the source and target domains are the same, while the robot dynamics models of the two domains are different. As shown in Figure 6, the successful transfer of the controller \(\mathbf{u}_{\text{s}}\) to the target domain \(\mathcal{D}_{t}\) highly depends on the control alignment. The  
Figure 5: The Data-informed Domain Randomization (DDR) adaptation mechanism.  
proposed transfer RL via DDR has been illustrated in Figure 6. During training, the trajectories \(\mathcal{T}_{s}\triangleq(\mathbf{x}_{s}(t),\mathbf{u}_{s}(t),\mathbf{x}_{s}(t+1))\) and \(\mathcal{T}_{t}\triangleq(\mathbf{x}_{t}(t),\mathbf{u}_{t}(t),\mathbf{x}_{t}(t+ 1))\) of the AUV were collected through interactions with the source and target dynamics, respectively. Based on the collected \(\mathcal{T}_{s}\) and \(\mathcal{T}_{t}\), the neural network \(H\), \(G\), and \(R\) are trained in the following order. The network \(H\) is trained and then fixed in training \(G\). Once \(H\) and \(G\) are obtained, \(R\) was trained.  
The control alignment depends on \(H:\mathcal{X}\times\mathcal{U}_{s}\mapsto\mathcal{U}_{t}\), which is learned together with \(G\). Different from existing research, these two mappings are learned in the DDR process. In the forward source dynamics model \(\mathcal{F}_{s}\) is trained to build \(H(\mathbf{x}_{s}(t),\mathbf{\tau}_{s}(t))=\mathbf{x}_{s}(t+1)\). The forward model takes in the state \(\mathbf{x}_{s}(t)\) and the control \(\mathbf{\tau}_{s}(t)\) at time \(t\) as inputs and outputs the state at time \(t+1\). The learning of \(F_{s}\) is conducted offline via supervised learning, based on the trajectories collected through numerical simulations. The loss function regarding forward model \(F_{s}\) is given as  
\[\min_{F}\mathcal{L}(F)=\sum_{(\mathbf{x}_{s}(t),\mathbf{\tau}_{s}(t),\mathbf{ x}_{s}(t+1))\in\mathcal{T}_{s}}\lVert\mathbf{x}_{s}(t+1)-F(\mathbf{x}_{s}(t), \mathbf{\tau}_{s}(t))\rVert. \tag{10}\]  
Given forward model \(F_{s}\), the loss function to train \(H\) is defined as  
\[\min_{H}\mathcal{L}(H)=\sum_{(\mathbf{x}_{t}(t),\mathbf{\tau}_{t}(t)),\mathbf{ x}_{t}(t+1))\in\mathcal{T}_{t}}\lVert\mathbf{x}_{t}(t+1)-F(\mathbf{x}_{t}(t),H( \mathbf{x}_{t}(t),\mathbf{\tau}_{t}(t)))\rVert. \tag{11}\]  
Then, keeping the \(F_{s}\) model fixed, the model \(R\) is trained. In addition, the transformation from \(\mathbf{u}_{t}\) to \(\mathbf{u}_{s}\) should be reversible. The translated control can be mapped back to the original domain. This requirement adds a regulation in training \(R\), so the loss function is given as  
\[\min_{R}\mathcal{L}(R)=\mathbb{E}_{\mathbf{\tau}_{t}\sim p(\mathbf{u}_{t})} \lVert H(\mathbf{\tau}_{s}(t),R(\mathbf{x}_{s}(t),\mathbf{x}_{s}(t+1)))- \mathbf{\tau}_{t}\rVert. \tag{12}\]  
Once these networks are trained, the target controller can be obtained as follows  
\[\mathbf{u}_{t}=H\{R[\mathbf{x},F(\mathbf{x},\mathbf{u}_{s}(\mathbf{x}))]\},\]  
and can be applied to the target dynamics \(\mathcal{F}_{t}\). When the real AUV system is subject to sudden changes, the mappings \(G\) and \(H\) are retrained. As shown by the results in Section 5, only a few data from various episodes are required.