18th International Conference on Innovative Computing, Information and Control (ICICIC2024)
Onsite (Qinhuangdao, China) and Online, September 10-13, 2024

PS-1: ICICIC2024-SS21-01
Research on Path Planning for Unmanned Surface Vessels Based on the Improved
Proximal Policy Optimization Algorithm
1

2,

2

Hongsong Zhao , Hongzhou Zhang * and Dawei Zhao
2

1
School of Information and Electronics, Beijing Institute of Technology, Beijing 100081, P. R. China
College of Intelligent Systems Science and Engineering, Harbin Engineering University, Harbin 150001, P. R. China*Corresponding author: zhz@hrbeu.edu.cn

IntroductionTo effectively exploit marine resources, advances in information technology and the need for ocean exploration have givenrise to the development of Unmanned Surface Vessels (USVs). As autonomous surface vessels, USVs play a key role in ocean
missions, and their unmanned control systems are at the core of advancing ship technology.
Global path planning for unmanned vessels involves the application of various algorithms to automatically avoid obstacles inorder to determine the best safe path in a map containing static obstacles such as reefs and islands. However, these algorithms
take too long to compute and require high computational power, so it has been proposed that local path planning or dynamic
obstacle avoidance focuses on real-time obstacle avoidance. In recent years, with the progress of science and technology, deep
reinforcement learning algorithms have made significant breakthroughs in the field of path planning. Deep reinforcement learningalgorithms have good decision-making ability and can explore and learn autonomously.
In this paper, an ICDM-PPO algorithm is proposed in order to improve the adaptability of collision avoidance algorithms to theenvironment and enhance the real-time planning ability of the algorithm. The main structure of this paper is as follows: firstly, weestablish a surface unmanned boat motion model to study the autonomous path planning capability, and secondly, we integrate
the Intrinsic Curiosity Distillation Module (ICDM) with the PPO algorithm, which drives the unmanned boat to explore the unknown
state through the curiosity satisfaction value, and at the same time adopts distillation method to improve the accuracy of stateprediction, and optimally adjusts intrinsic rewards by taking advantage of the uncertainty in the frequency of state exploration. TheICDM-PPO algorithm is formed. The algorithm combines strong exploration ability and high prediction ability to further improve
the learning efficiency.

Research Questions

Methodologies

In the field of reinforcement learning, the main
goal pursued by an Agent is to maximize the reward
it receives through environmental feedback.
However, in the face of sparse reward
environments, it becomes a great challenge toconstruct a reward system that is both efficient anddense, which often leads the Agent to fall into a local
optimum. In the unmanned boat path planningproblem, the Agent obtains rewards and iteratively
optimizes its strategy by interacting with theenvironment, but this interaction relies on blindstochastic exploration until the goal is reached.

We propose an approach that combines the curiosity
mechanism and the distillation module, which provides additional
intrinsic incentives to facilitate the Agent’s exploration of unknown
environments when it encounters unexplored states. In addition, byfusing intrinsic and extrinsic rewards, the approach enables theAgent to learn and explore more efficiently, avoiding simply pursuingnovelty states while neglecting the efficiency of goal-directed
learning. Secondly, the intrinsic curiosity distillation module isintegrated with the PPO algorithm (ICDM-PPO algorithm), which
employs the distillation method to improve the accuracy of stateprediction and optimally adjusts the intrinsic rewards by exploiting
the uncertainty of state exploration frequency.

Tables (1)Table 1. Parameter settings for simulation experiments
Parameters
Maximum number of iteration steps per round step
Maximum number of rounds episodeActor E-learning rate αCritic E-learning rate β
Discount factor γBatch-size
Truncation factor ε
GAE parameter λICDM learning rate
ICDM proportionality factor η
ICDM proportionality factor μ
ICDM loss factor

Value
1000
3000
0.001
0.001
0.9
64
0.2
0.95
0.001
100
100.2

18th International Conference on Innovative Computing, Information and Control (ICICIC2024)
Onsite (Qinhuangdao, China) and Online, September 10-13, 2024

Figures(a) ICDM-PPO algorithm path diagram(b) PPO algorithm path diagram

Figure 1. Comparison of paths for simulation experiment(a) Incentive change curve(b) Step change curve

Figure 2. Training effect graph of ICDM-PPO algorithm(a) Incentive change curve(b) Step change curve

Figure 3. Training effect graph of PPO algorithm

Tables (2)Table 2. Specific data of simulation experiment
Algorithm
ICDM-PPO algorithm
PPO algorithm

Path length
1299.11453.25

Step85
unstable

Rewards
0
unstable

Success rate
63.3%
37.6%

Conclusion
The full paper investigates the autonomous path planning capability of a surface unmanned boat in a static and complex
environment containing special obstacles. A simulation map environment was created using Python’s Tkinter module andPyTorch was applied to construct a deep neural network model as a means of evaluating the autonomy and intelligencecapabilities of the unmanned boat. By integrating the ICDM module with the PPO algorithm, the ICDM-PPO algorithm wasformed and validated in a specific environment with long obstacles. The experimental results show that compared with the PPO
algorithm, the ICDM-PPO algorithm is able to guide the unmanned boat to avoid wandering and plan a shorter and smoother
path more effectively in the special obstacle environment. In future work, collaborative multi-unmanned craft path planning canbe investigated based on the results of this paper.