Proceedings of the Thirty-second (2022) International Ocean and Polar Engineering Conference
Shanghai, China, June 5–10, 2022
Copyright © 2022 by the International Society of Offshore and Polar Engineers (ISOPE)
ISBN 978-1-880653-81-4; ISSN 1098-6189

www.isope.org

Deep reinforcement learning-based controller for path following of deep sea mining vehicle
Qihang Chen1, Jianmin Yang1, 2*,Chang Zhou1, Jinghang Mao2, Changyu Lu1
1 School of Naval Architecture, Ocean & Civil Engineering, Shanghai Jiao Tong University
2 Yazhou Bay Institute of Deepsea SCI-TECH, Shanghai Jiao Tong University1Shanghai, China 2Sanya, Hainan, China

problem of path following for ground walking robot. Yeu et al. (2012)
and Yoon et al. (2014) combined the following algorithm with thepositioning of DSMV, proposed the traditional path following algorithmand designed a PI controller to realize speed control. On this foundation,
to make DSMV show more stable speed and steering ratio control
performance in the process of path following control, subsequent
researchers have made some optimization design and improvement. In
2015, Yeu et al. improved the PI controller and proposed theApproximately Millisecond Constrained Integral Gain Optimization(AMIGO) rule with validation of this model. Dai et al. (2019) further
established the dynamic models for the DSMV, adopted fuzzy control
algorithm and fuzzy adaptive PID control algorithm to control track
speed and speed ratio separately. This work solves some shortcomingsof the PI controller.
Besides speed control, heading control and slip control has also attracted
the attention of many researchers. Varshney et al. (2013) proposed aDSMV speed and heading angle control method, focusing on DSMV
perception and attitude control by considering drag characteristics, but
still based on PID controller. Ouyang and Wang (2016) proposed animproved sliding mode control algorithm to track the optimal track slip
rate by controlling the torque of the driving wheel and demonstrated its
good performance by simulation. Liu and Guo (2018) proposed a novel
trajectory tracking PID control algorithm. Through continuous
correction of the heading angle and track deviation feedback, thedesigned goal of the DSMV path following control was achieved.
Most of the existing research on control methods are based on traditional
PID control method, and the improvement of robustness and adaptability
mainly depends on the combination of PID controller, fuzzy algorithm
and adaptive algorithm. Although the control design requirements are
satisfied, the complex parameter adjustment process and the results thatare not easy to promote limit the pace of the controller towards a moreaccurate, efficient, safe and green direction. To solve the problem, weneed to find possible tools from other fields which are more suitable for
DSMV path following control to satisfy the high requirements of thefuture mining operations.
Luckily, the latest development of artificial intelligence (proposed byLiu et al., 2017) has had a far-reaching impact on the industrial world,which has brought powerful algorithms for researchers to describe andcontrol extremely complex systems in a changing environment.
Reinforcement learning (RL) is an epitome of artificial intelligence

ABSTRACTThe relevant research of deep-sea mining system is the active area of
research of deep-sea resources development, among which the pathfollowing performance of deep-sea mining vehicles (DSMV) is directly
related to the mining efficiency and operation safety of the whole mining
system. In this paper, a deep reinforcement learning (DRL)-based
controller for DSMV is trained and verified. An actor-critic-based
reinforcement learning algorithm for continuous action space problems
is adapted. A Markov decision process (MDP) model including the state,action, and reward is designed. The proposed method’s path-following
and self-learning capabilities are validated through DSMV pathfollowing simulation test.

KEY WORDS: Deep-sea mining vehicle, deep reinforcement
learning, TD3, path following, simulation, Markov decision processmodelINTRODUCTION
The ocean, which accounts for 71% of the earth's surface area, is thelargest untapped mineral resource on the earth. In generally, there arethree main types of deposits of deep-sea mineral resources on the sea
floor (reviewed by Leng et al., 2021): polymetallic nodules (also termed
as manganese nodules, resting up to 4000–6000 m water depth), massive
Sulfides (SMS, located near hot water vents on volcanic rocks) andcobalt-rich crusts (found on the slopes and summits of undersea
mountains). Once this rich mineral resource can be exploited effectivelyand economically, it will alleviate the shortage of mineral resources.
As the key equipment of deep-sea ore mining in the future, deep-sea
mining vehicle (DSMV) is the carrier of mining equipment. DSMV
carries mining devices to collect deep-sea ore as much as possible
through the ore area, and its ability to follow the predetermined mining
path directly influences the effective of the ore collecting device in themining process, which leads to the change of mining efficiency of thewhole system. To improve mining efficiency and ensure operation safety,
researchers have been working to find more efficient, accurate and robust
path following controller.
Rankin et al. (1996) proposed the follow-the-carrot method to solve the108