# Continuous Autonomous Ship Learning Framework for Human Policies on Simulation
## 2 Related Works
### RL for a Hierarchical Autonomous Ship Task  
Autonomous ships performing missions in the ocean instead of human beings have been investigated in various fields. The control and route search of a ship began with the navigation of a ship to a place where human beings wanted to go by controlling the direction and power based on the natural environment, including wind, waves, and tides. Ship navigation was approached using the physics and control algorithm based on a mathematical algorithm [42; 43; 44]. The A* algorithm was also adopted to plan the route to the waypoint, given destination in the given space, clearance of noise collected through sensors, interference and slipping of a ship in the ever-changing natural environment, and real-time adaptive ship control using Gaussian process regression [45; 46; 47]. RL has emerged as an important technology by replacing complicated mathematical algorithms and solving problems. It has been demonstrated to have high potential for applications in ship movement control, posture control against the natural environment, collision avoidance, and path finding [8; 9]. Studies on autonomous ships using the RL algorithm in the control field were applied to obstacle avoidance, decision on access to targets, speed control, and posture control through value-based Q-learning [48]. The algorithm was also investigated in the ship control field by applying a policy-based deep deterministic policy gradient (DDPG) [49]. Moreover, studies have been conducted on learning the posture control model of a ship in real-time by configuring the given route based on the DDPG and tracking the configured route [21]. The studies above dealt with the movement issue of a ship in the local space to consider the global space or surrounding conditions depending on the necessity, focusing on the ship's posture. Accordingly, the local space in the ship control field can be defined by the natural environment, which affects the ship control and present state of a ship. Therefore, the supervision space and obstacle avoidance that must consider the given global space can be separated from the local space. Furthermore, studies on autonomous ships using the RL algorithm in the navigation field were applied to Q-learning-based smart ships, path finding, and ship control. The environments (including the distance to a destination and punishment on obstacles) and restricted areas were implemented based on the Nomoto model in a narrow waterway. Accordingly, autonomous ships learned human-defined rules in those environments [25]. The RL algorithm was also applied to decision-making on autonomous navigation under partial observation considering a real environment [50]. The multi-agent algorithm was applied to maintain the formation during navigation and path finding, in which several autonomous ships sailed [51]. Studies on the navigation section aimed at the movement of a ship to a destination in a given marine space. Accordingly, they focused on the acquisition of state data of a space through sensors, route planning considering other ships, and obstacle avoidance. Similar to the studies on the control field, studies on the navigation sector also included the control field depending on the circumstances. The curse of the RL algorithm dimensionality can be cleared by separating complicated issues to be solved by autonomous ships, as described above. Through stratification, the knowledge domain per problem can be separated, and the Markov decision process (MDP) of the hierarchical structure can be configured [52]. The most significant merits acquired from the stratification are to save the learning time for actions and control of the autonomous ship and to demonstrate definite learning results [53]. The hierarchical structure enables supervised/non-supervised interlayer learning and has the advantage of learning speed and generalization by simplifying the algorithm [54].