# Improved DQN for Dynamic Obstacle Avoidance and Ship Path Planning
## 1 Introduction  
With the rise of artificial intelligence, developing the intelligence of transportation vehicles has become one of the essential objectives of artificial intelligence research and exploration. The intelligent navigation of ships is of great practical significance for improving the economic benefits of shipping enterprises. The development of artificial intelligence technology, especially reinforcement learning and its related methods, has complied with the requirements of intelligent ship collision avoidance and intelligent navigation [1; 2]. Reinforcement learning emphasizes the agent's learning from the environment to behavior mapping, seeks the correct action decision by maximizing the income, and combines the profit and loss considerations in the operation process, which is a more appropriate method system in the intelligent collision avoidance of ships. Therefore, the essence of intelligent collision avoidance under the reinforcement learning system is to allow the agent to make the sequential decision to maximize the benefits under the balance of profit and loss. The function of ship path planning is to complete the navigation index of the current and target position of the ship under a restricted navigation area and among obstacles, combined with the external environment and ship conditions. Obstacle avoidance is to avoid obstacles in an unknown dynamic environment during navigation. During driving, the operator will adjust the local path of the ship. The purpose of the traditional method is to provide a complete deterministic solution at the decision-making level, including manual rules and automatic execution, by analyzing the fusion information obtained by the ship at each decision-making time. At the core of the traditional method is the decision rules that people make according to the problem situation. [PERSON] et al. [3] used the Fast Marching Method (FMM) to generate collision avoidance paths for the USV in a dynamic environment to ensure the safety of the USV. [PERSON] et al. [4] combined A * and B spline methods to generate a feasible path without obstacles. [PERSON] et al. [5] described a navigation guidance and control system based on LOS and applied it to uncrewed ships considering only staticobstacles. [PERSON] et al. [6] proposed a collision avoidance method for USV based on Rapidly-exploring Random Tree (RRT) and conducted a simulation verification on the MATLAB platform. [PERSON] et al. [7] proposed an improved neuro-fuzzy algorithm to construct the collision avoidance system of USV. When the constraints become more complex and the variable dimension increases, the traditional path planning algorithm has some defects, such as high time complexity, and quickly falls into local optimization. Reinforcement learning [8; 9] has developed rapidly in intelligent navigation because of its simple structure and strong adaptability. It was first proposed by [PERSON], whose essence is to learn from the interaction. In recent years, there has been a boom in theoretical research and technical implementation of reinforcement learning, especially in the Deep Q-Network (DQN) algorithm [10; 11] proposed by the Google team. In the shipping industry, reinforcement learning is widely used in collision avoidance. By integrating the navigation prior knowledge to design the ship's perception state and reward function, the action space can be designed based on the course control, and the DQN method can be applied to the ship collision avoidance. Deep learning has a strong ability for data analysis and perception. It can obtain helpful knowledge from multimodal information to strengthen learning without prior knowledge. Through the interaction between the agent and the environment, it can learn the best strategy to maximize the accumulated income. [PERSON] et al. [12] proposed an intelligent path planning method based on deep reinforcement learning, which solved the problem of vehicle tracking error and over-dependence in traditional intelligent driving vehicle path planning. [PERSON] et al.