# An Enhanced Deep Q Network Algorithm for Localized Obstacle Avoidance in Indoor Robot Path Planning
## 2 Related Work
Reference [21] cleverly combines the RRT strategy to address the challenge of Double DQN failing to learn new experiences, while also integrating A\({}^{\star}\) algorithm ideas into the reward mechanism to tackle the model convergence issue. Reference [22] combines Hindsight Experience Replay with Double DQN to mitigate the negative impact of sparse rewards. Both studies demonstrated the effectiveness of their algorithms in a 2D grid map. However, it remains uncertain whether these algorithms can be effectively applied in unknown environments. References [23; 24; 25; 26; 27] combine DQN with APF, utilizing the obstacle avoidance capability of APF and the learning ability of DQN to improve the path planning effect of mobile robots or drones in complex environments, thereby improving efficiency and success rate and addressing the shortcomings of traditional methods in complex dynamic environments. Ref. [23] proposed DM-DQN, which combines the dueling network and Munchausen strategy, and designed a reward function based on APF. Reference [24] uses APF as prior knowledge, introduces the SA-\(\varepsilon\)-greedy algorithm to adaptively adjust the random exploration frequency, and uses a multi-output neural network and B-spline algorithm. Reference [25] optimized the path planning of UAVs in urban environments by improving the action space and reward mechanism. Reference [26] improved the robot's navigation ability in static and dynamic obstacle environments by introducing APF into the reward function. At the same time, the knowledge transfer method was used to migrate the training model in a simple environment to a complex environment, which accelerated training and convergence. These algorithms solve the shortcomings of traditional methods in complex environments by combining the obstacle avoidance of APF and the learning ability of DQN. However, these methods still face generalization and stability issues in extremely complex and dynamic environments, which require further research and optimization. The research in references [28; 29; 30; 31] all focuses on the path planning problem of UAVs in three-dimensional environments, and is committed to optimizing the autonomous navigation and obstacle avoidance capabilities of UAVs in complex terrain and obstacle environments. Ref. [28] proposed a Retrospective-Based Deep Q Learning (R-BDQL) method,which reduces the overestimation problem in Q-value estimation by introducing a retrospective mechanism. For extremely complex environments, it may take a lot of time to optimize the algorithm, and in certain cases, overfitting problems may occur. Ref. [30] Combining the heuristic function with the maximum average reward experience replay mechanism improves sample utilization and algorithm convergence speed. However, in large-scale dynamic environments, the heuristic information may not be accurate enough, affecting the path planning effect. Ref. [31] introduced a Memory-Enhanced (ME) mechanism combined with the Duel DQN to effectively reduce collision events in early training. However, in extremely complex or dynamic environments, the ME may lead to low memory utilization and fail to adapt to environmental changes. To sum up, DRL is often used to address local path planning and real-time obstacle avoidance challenges. According to its implementation method, RL can be divided into two categories: model-based and model-free [32; 33]. Model-based methods, such as Q Learning, rely on explicit environmental models, such as state-action tables. Model-free methods do not require detailed information about the environment and learn the best strategy through exploration. These model-free methods can be further divided into value-based learning algorithms and policy-based learning algorithms. Value-based methods train agents by evaluating the results of performing specific actions in specific states, while policy-based methods directly construct a mapping strategy from state to action [34; 35; 36].