# Obtaining Human Experience for Intelligent Dredger Control: A Reinforcement Learning Approach
## 6 Conclusions  
The operation of CSDs is affected by various uncertainties such as the type of soil, the terrain of the water bed, as well as the measurement errors of instruments. Even for an experienced human operator, it is hard to manipulate the cutting process in a stable and efficient manner. In this work, we focus on a learning approach to the sequential decision-making problem of the cutting process. Since the dredging environments contain various uncertainties, it is intractable to design a static model to describe the dynamics of the cutting process. Thus, the proposed learning approach combines a neural network model with a RL model. The neural network model is supposed to construct a concise a virtual environment based on the data collected during the time when an experienced human operator manipulates the CSD. Afterwards, the RL model can learn the optimal control policy by interacting with the virtual environment, without worrying about accidents or mistakes in the trial-and-error learning period. The results show that the proposed learning approach can gain the experience of human operators from historical data, and, in addition, can outperform the experienced human operator with respect to the production, taking account of various operational constraints.  
Figure 7: Production comparison between an experienced human operator and three intelligent control models (i.e., Q-learning, SARSA, and SARSA(\(\lambda\))).  
It should be noted that after several hours of learning from the data of a single operator, the RL agent cannot build up a perfect mode to cope with all future uncertainties. However, when the RL agent can imitate the behavior of an experienced human operator within a virtual environment, we can allow it to directly interact with a real-world scenario to further improve the model. In this work, we cannot allow the RL agent to learn from scratch in a real CSD because of the safety reasons. Once the learning agent is affordable to control a real CSD, we can allow it to improve itself through trial-and-error interactions in the real-world scenario, which will be investigated in our future work.  
Future work will also investigate the RL approach in continuous state space and action space, as well as function approximation methods to store the state transitions, etc.. Moreover, we will look at the model-based RL approach to this problem, in which the environmental model can be built up from samples, and then planning (instead of model-free learning) methods can be used to find the optimal action for each state based on the learned environment model. Lastly, we would like to evaluate the proposed approaches in a real CSD, and it is also our ultimate goal to release human operators from the boring but daunting mission of manipulating the cutting process of a CSD.  
Conceptualization, C.W. and F.N.; methodology, F.N.; software, X.C.; writing--original draft preparation, X.C.; writing--review and editing, C.W.; supervision, F.N.; funding acquisition, C.W.  
This research is supported by the National Natural Science Foundation of China (Grant No. 61703138), and Natural Science Foundation of Jiangsu Province (Grant No. BK20170307).  
The authors declare no conflict of interest.