# Obtaining Human Experience for Intelligent Dredger Control: A Reinforcement Learning Approach
## 1 Introduction  
The Cutter Suction Dredger (CSD) is a special vessel designed for the maintenance of rivers, lakes, or ports. The CSD can dredge nearly all kinds of soils (sand, clay, rock) into fragments with a cutter head, and then the dredged materials will be pumped into a pipeline and transported to the discharge zone. Currently, the CSD is usually controlled by human operators, and it is a daunting challenge for them to maintain a good production rate during the so-called cutting process. This is because the geological conditions of the working site are uncertain and dynamic for the operators. In other words, the operators cannot have prior knowledge about the types of soils to be dredged, so they must concentrate on observing instrument changes in real time during the cutting process. Operating the CSD requires that the slurry density in the pipeline cannot be too low. However, it is also dangerous to maintain a high slurry density because the pipeline can be blockage [1; 2; 3]. As a result, long-term manual operation often causes driving fatigue, resulting in operational accidents and inefficiencies. To reduce the labor intensity of the operator, in this work we aim at developing an intelligent cutter controller that can replace the human operators to manipulate the cutting process of the CSD. The operation of the CSD need to continuously manipulate several motion controls, such as the cutter or swing control, the spud carrier control, the anchor control, and the pump control. Those motion controls need to be geared to one another during a full dredging cycle. In this work, we are in particular interested in the _cutter or swing control_ ( named as the cutting process throughout this work) because it directly determines the production rate of a CSD. Moreover, the cutter control is the most boring but daunting mission for human operators, compared to other motion controls. Thus, an intelligent cutter controller is the decisive processor to release the operators from heavy workload. To design an intelligent controller for the cutting process is not a trivial task. The major challenge is that the geological conditions of the working site are uncertain and dynamic. Thus, it is intractable to build up a precise model that can be applied to handle a diversity of geological conditions. Learning is an effective means of making real-time responses to the environment through trial-and-error interactions. Among the learning methodologies, Reinforcement Learning (RL) can obtain the optimal control policy for sequential decision-making with little prior knowledge [4; 5]. It has been shown that the agents in many RL tasks can successfully achieve a goal or win a game, e.g., Atari games [6], robots [7], the game of Go [8], and fighting game Street Fighter [9]. For a learning agent to imitate human behavior in controlling the cutting process, it must understand the intentions or objectives of the operators. However, in real-world industrial problems, we cannot easily find a win or lose mechanism as in games to repeat the learning episodes. Moreover, since the terminal state is not obvious, we must figure out the task objective of the cutting process based on the historical dredging data produced by experienced human operators. Thus, we need to analyze the cutting process to understand what to achieve regarding a RL task. Unlike many robot leaning tasks [10; 11; 12] and strategy learning tasks [13; 14], we cannot allow the learning agent to directly make trial-and-error interactions with real dredging environments so as to lean the optimal behavior. This is because it is dangerous for an inexperienced controller to explore the action space in practical dredging projects, and controlling a real CSD to make trial-and-error interactions is unnecessarily expensive. In this work, our learning approach will construct a virtual environment based on historical dredging data. Those data are collected during experienced human operators carry out actual dredging projects.