# Continuous Autonomous Ship Learning Framework for Human Policies on Simulation
## 3 Intelligent Learning Framework for Autonomous Ships
### Architecture  
The learning framework (ILFAS) proposed in this study comprises three types: hierarchical RL frame to be applied to autonomous ships, case-based curriculum system-induced learning of autonomous ships, and environment used in the learning. The physical environment was created in three dimensions (3D) with a fixed height for the LiDAR simulation using unity. Moreover, the basic physics model for the acceleration of ship weight and surface sliding in the ocean was adopted.  
The elements are explained from left to right in Figure 1 to facilitate the understanding of the learning frame.  
The hierarchical learning structure comprises the autonomous mission (_Mission_), which reduces the learning complexity of the autonomous ship to learn, autonomous sailing (_Sailing_), and autonomous control (_Control_) to make decisions and operate various vessels. Considering _Sailing_ in Figure 1, the autonomous ship was learned through transfer learning using a case-based curriculum system, and _Control_ was learned through supervised learning based on _Sailing_. After _Sailing_ was learned, \(Sailing(a,s_{t},s_{t+1},r)\) at which the autonomous ship should move to _Control_ could be presented. Therefore, the expected action value \(a\) (the direction and speed) by _Sailing_ could also be predicted. _Sailing's_ action value \(a\) and current state _s\({}_{t}\)_ are the _Target_ and _State_ values of _Control_, which were provided along with environmental data, such as wind. Through _Control_'s action in the learning process, the autonomous ship operated in the environment. The operation result of the autonomous ship was obtained as _s\({}_{t+1}\)_ of _Sailing_, and _Control_ learns by returning the difference from the expected action value to _Control_ as reward \(r\). _s\({}_{t}\)_ is environment data through _Sailing_ and environment plug-in learning become the state and target value of _Control_. The state was transferred to _Control_, the autonomous control, considering learning on _Sailing_, and the actions from _Control_ are evaluated and learned by the autonomous ship. _Mission_ is a general RL field related to space planning. For example, the marine space was divided based on the supervision scope of autonomous ships for ocean surveillance. The division of the marine space and movement location in sequence was determined for collecting marine data. A destination was delivered to _Sailing_ through the spatial data input from the environment, and the learning results of _Sailing_ were evaluated based on the successful arrival to the destination.  
A case-based curriculum system configured the curriculum with the degrees of difficulty and space by defining cases of neighboring ships to be operated in the ocean. The curriculum was classified into the global curriculum that learns the entire given space and the local curriculum based on the sequence of time and space. The curriculum was provided in the environment in real-time. The cases stated in the curriculum were provided based on the decision-making order of neighboring ships. Moreover, the RL algorithm implementing the cases already learned the movement between waypoints.  
Figure 1: ILFAS scheme. Autonomous ship, hierarchical RL algorithm, case-based curriculum system building the curriculum and operating neighboring ships through cased-based RL, 3D geographical environment with a certain height and learning environment integrate the elements above. The environment and plug-in can be replaced by other simulations.