# Continuous Autonomous Ship Learning Framework for Human Policies on Simulation
## 3 Intelligent Learning Framework for Autonomous Ships
### Hierarchical RL Frame  
This study hierarchically configured the RL of an autonomous ship to enable its operation in reality. The correlation between separate layers was explained using _Sailing_ and _Control_ as examples. RL based on a multi-agent algorithm was applied to _Sailing_ for learning autonomous navigation, including path finding, collision avoidance, and human-defined navigation rules considering the operation of other ships. _Control_ learned the actions \((a_{s})\) to control the ship based on the environment transferred from _Sailing_. This process is consistent with the control methods and types of ships. The RL algorithm can be applied to existing ship and posture controls. The state input and action output of _Sailing_ are explained. \[(o_{s}\in O)=\Big{\{}\Big{[}Self_{(Position,\,Speed,\, Direction,\, Others)}\Big{]},[Env_{all}],\,\,[G_{Goal\,\,Pos,\,Dir}],\,\,\Big{[}O_{Obstacle\,\,by\,\, Lidar}\Big{]}\Big{\}} \tag{1}\]  
\((o\in O)\) observed in the autonomous ship for the decision-making of _Sailing_ is defined in Formula (1). \(Self\) is the ship state; \(Env\) is transferred from the environment, and it includes the data from the natural environment simulator. The absolute coordinates and relative directions of destination \(G\) were calculated. The LiDAR of \(O\) is the distance between the static obstacles, such as an embankment. The data were acquired through the short-range obstacle detection in the environment and dynamic obstacles (the neighboring ships operating around). \[(a_{s}\in A)=\{[Speed\,\,up,\,Keep,\,Speed\,\,down],\,\,[Left,\,Keep,\,\, Right]\} \tag{2}\]  
The action \((a_{s})\) was discretely declared to learn _Sailing_. The maximum scope that can be changed at a time using Formula (2) limits the scope of one output action, such as \((10\,\,knot,\,5\,degree)\). The scope to be changed at a time varies based on the vessel type. The scope is the criteria for evaluation and compensation of the actions of _Control_. \(\Big{[}O_{Obstacle\,\,by\,\,Lidar}\Big{]}\), the single-layer LiDAR inputs the obstacle detection data around the autonomous ships, and other sensors can be added when required. The relative location value of the autonomous ship was calculated using the obstacle and destination data, such as the real environment. The network of autonomous ship layers for path finding is defined, as shown in Formula (3). Considering the reward on _Sailing_, when the autonomous ship successfully arrives at the destination provided in the mission, the goal and collision are one and 0.001, respectively. The depreciation rate (\(\gamma\)) is 0.0001. The state value is received from the environment. The _Sailing_ network is shown in Formula (3):  
\[[Network\,\,\pi_{s},\,\,Replay\,\,Memory\,\,D_{s},\,\,Parameter\,\,\theta_{s},\, \,Goal\,\,G_{s},\,\,Reward\,\,R_{s},\,\,Action\,\,a_{s}] \tag{3}\]  
After learning, the _Sailing_ layer is ready to perform supervised learning for _Control_. The observation value of _Control_ is the state value of _Sailing_ and is defined, as shown in Formula (4). \(Self_{(Others)}\) is the present state value, including the gradient of a ship. \(Env_{(Option)}\) is selectively applied based on the natural environment simulation, such as a plug-in. This value is the vector value of the wind direction and tide. Its relative value to an autonomous ship is calculated as shown in Formula (4):  
\[(o_{c}\in O)=\Big{\{}\Big{[}Self_{(Position,\,Speed,\, Direction,\, Others)}\Big{]},\Big{[}Env_{(Wind,\,Wave,Tidal\,\,Direction|Option)}\Big{]}\Big{\}} \tag{4}\]  
\(Env\), the natural environment, was provided through _Sailing_, and the network is shown in Formula (5). The action \((a_{s})\) determined in _Sailing_ is defined as the target value \((task_{target}=\pm 10\,\,knot,\,\,\pm 5\,\,degree)\) of _Control_. If \(a_{s}=(Keep)\), then the value zero (\"0\") is transferred. The target value of _Control_ is given as _Goal_\(a_{s}\), which is the action to be taken by _Sailing_.