# Continuous Autonomous Ship Learning Framework for Human Policies on Simulation
## 3 Intelligent Learning Framework for Autonomous Ships
### ILFAS Training  
Although the types of RL algorithms have been continuously developed, they should prevent local optimization and maximization of rewards. Other researchers frequently use separately designed reward functions to achieve the above goals quickly. However, this study aims to train an RL algorithm designed to provide generally delayed and immediate rewards. The subsequent algorithm is the policy-based multi-agent RL COMA algorithm. The following formula demonstrates the policy gradient with the Q function as the advantage using the actor-critic. The actor updates policies and the critic evaluates them in this configuration. Policies are updated simultaneously, and rewards are maximized as expressed below:  
\[\
abla_{\theta}J(\theta)=E_{\tau\sim p_{\theta}(\tau)}[\sum_{t=0}^{T}(\
abla_{ \theta}log\pi_{\theta}(a_{t}|s_{t})A^{\pi_{\theta}}(s_{t},a_{t}))] \tag{12}\]\[A^{\pi_{\theta}}(s_{t},a_{t})=(Q^{\pi_{\theta}}(s_{t},a_{t})-V^{\pi_{\theta}}(s_{t})) \tag{13}\]  
The RL applied to multiple ships updated \(log\pi_{\theta}(a_{t}|s_{t})\), the actor part taking log using Formula (9), and critic \(A^{\pi_{\theta}}(s_{t},a_{t})\) comprising the Q function. Policies and rewards were each updated as critics in Formula (12), as shown in Formula (13), and approximated to \(V^{\pi_{\theta}}(s_{t})\approx V_{w}(s_{t})\):  
\[\theta\leftarrow\theta+\ \alpha_{actor}\
abla_{\theta}J(\theta) \tag{14}\]  
\[\mathrm{w}\leftarrow\mathrm{w}+\ \alpha_{critic}\
abla_{w}\mathrm{V}_{w} \tag{15}\]  
The actor and critic performed transfer learning by learning the environment changed based on Formulas (14) and (15). This study configured the curriculum, and Definitions (8) and (9) are explained in detail as follows:  
Global Curriculum = [Find entire route], [avoid fixed obstacles], [case: avoid dynamic obstacles], etc. Local Curriculum = [Stand by at dock area], [passing method in waterways], [avoid water recreation zone in coastal waters], etc. The local curriculum in learning solves more difficult issues, although the global curriculum environment continuously exists. Regarding the RL of autonomous ships interacting with dynamic obstacles, path planning for autonomous navigation is learned in the global curriculum, and the navigation rules by collision avoidance are learned in the local curriculum. When _Sailing_ learning is completed, as shown in Figures 2 and 3, _Control_ learning starts. The following example illustrates a curriculum configured for autonomous ships. Figure 2: Schematic of the inter-layer supervised learning of hierarchical RL. According to the goals defined by humans, the _Mission_ layer analyzes the space and sets the destination (left). The set goal is transmitted to the _Sailing_ layer, and a delay reward is made according to the result. Sailing learns navigation as in Formula (3) according to Formula (1) observed from the environment, and the route consists of the actions as in Formula (2) (center). After learning, the _Sailing_ layer becomes an observation value of _Control_, as shown in Formula (4), and _Control_ learns, as shown in Formula (5), and is executed in the environment through the output Action \(a_{c}\). In the _Sailing_ layer, the control is compensated by comparing the learned target value with the environmental change caused by _Control_’s action. Transfer learning is based on baseline training, including pathfinding without obstacles. After learning was completed, learning was adjusted through (_Policy Iteration_), as shown in Formula (16). Adjustment applies the probability rate \(r(\theta)\). \[r(\theta)=\frac{\pi_{\theta}(a|s)}{\pi_{\theta old}(a|s)} \tag{16}\]  
After the policies were adjusted through transfer learning, as shown in Formula (16), the reward trajectory \(Trajectory(R(\tau))\) was generated by _Policy \(\pi_{\theta}\)_ with _Policy parameter \(\theta\)_, which consists of \(\{S_{1},A_{1},R_{2},S_{2},A_{2},R_{3},S_{3},A_{3},\dots,S_{n}\}\). After global learning, a new environment with time and space was presented to the autonomous ship through the local curriculum.