Proceedings of the Thirty-second (2022) International Ocean and Polar Engineering Conference
Shanghai, China, June 5–10, 2022
Copyright © 2022 by the International Society of Offshore and Polar Engineers (ISOPE)
ISBN 978-1-880653-81-4; ISSN 1098-6189

www.isope.org

Autonomous Underwater Vehicle Path Planning withHybrid Policy Based Policy-Constrained Deep Reinforcement Learning
Qiongxiao Liu 1 , Ben Li 1 , Yue Jin 2 , Zhemin Huang 1 , Guohua Xu 1∗
1

School of Naval Architecture and Ocean Engineering, Huazhong University of Science and Technology, Wuhan, Hubei Province, China
2
Department of Electronic Engineering, Tsinghua University, Beijing, China∗
Corresponding author

ABSTRACTenvironment and re-plan path periodically, which is prohibitively timeconsuming and unacceptable in urgent tasks(Chowdhury and Schwartz,2020). Additionally, marine environment is complex and dynamic,
which leads to great difficulties to build an explicit environment map andthus results in the inefficiency or even infeasibility of the conventional
methods.

Autonomous underwater vehicle (AUV) path planning is an importantproblem. To solve the problem, conventional methods require explicit
environment model. However, marine environment is complex andhighly dynamic, which leads to great difficulties for building an explicit
environment model and thus results in the inefficiency or even infeasibility of the conventional methods. In this paper, we propose a novel
deep reinforcement learning based method to solve the AUV path planning problem. Specifically, we focus on the AUV path planning problem
where the environment map is unknown and the AUV is required to reacha target point with the minimum time and obey motion constraints including both collision avoidance and steering angle minimization. We modelthe problem as a Markov decision process and propose a novel Hybrid
policy based Policy-Constrained PPO (HyPoC-PPO) algorithm to solve
the problem. HyPoC-PPO can directly guide the learned policy to minimize the steering angle of the path with a new objective function thatincorporates policy regularization rather than depends on indirect guidance from a reward function. In addition, HyPoC-PPO can generate ahybrid policy that is composed of DRL-based and non-DRL-based policies. The two kinds of policies are applied to different situations and canbe integrated well by optimizing our proposed objective function. Simulation results demonstrate the capability and efficiency of our proposed
method for AUV path planning in unknown environments.

In this paper, we propose a novel deep reinforcement learning based
method to solve the AUV path planning problem. Specifically, wefocus on the problem where an AUV is required to both reach a target
point in an unknown underwater environment with the minimum timeand circumvent obstacles around it with the least cumulative steering
angle. We model the problem as a Markov decision process (MDP).
The ranging results of a multi-beam forward looking sonar of the AUV
are regarded as the state of the MDP. The direction of a path segment isregarded as the action of the MDP.
To solve the MDP, we propose a novel policy-constrained deep
reinforcement learning (DRL) algorithm, i.e., Hybrid policy based
Policy-Constrained Proximal Policy Optimization (HyPoC-PPO).
Proximal Policy Optimization (PPO) (Schulman et al., 2017) is a
general DRL algorithm, which is popular in recent works due to its
sample efficiency and simplicity. However, for the AUV path planningproblem, vanilla PPO may not generate a path with the smallest
steering angle efficiently. Therefore, we adopt a policy regularization
method, which is based on both the difference between the directionsof two adjacent path segments and the absolute direction of eachpath segment to constrain the learned policy. The policy regularization can provide a direct guidance for the learned policy to generate
path segments both towards the target and with minimum steering angles.

KEY WORDS: AUV path planning; unknown environment; deep
reinforcement learning (DRL); proximal policy optimization; policyconstrained DRL; hybrid policy.

INTRODUCTION
Autonomous underwater vehicle (AUV) path planning is an important
problem in many applications, such as underwater exploration, oceandevelopment, underwater rescue, etc. To solve this problem, conventional path planning methods require an explicit environment map basedon which a path planning algorithm can be executed online(Zhanget al., 2018; Yan et al., 2021). When the environment is unknown, thesemethods need to gradually build a map according to the perception of theIn addition, HyPoC-PPO adopts the hybrid policy model proposed byJin et al. (Jin et al., 2019, 2021) to accelerate learning. The hybrid
policy model consists of a DRL-based policy and a non-DRL-based
policy. When an AUV observes an obstacle in the target direction,
the DRL-based policy is activated to plan a path enabling the AUV
to circumvent the obstacle. Otherwise, the non-DRL-based policy is1022