# Applying Reinforcement Learning for AMR's Docking and Obstacle Avoidance Behavior Control
## 2 Deep Q-Network Methodology
### DQN (Deep Q-Network)  
The Deep Q-Network (DQN) extends traditional Q-learning by leveraging deep learning to approximate the Q-value function, allowing it to handle high-dimensional state spaces more effectively. As illustrated in Figure 3, Q-learning becomes computationally \begin{table}
\begin{tabular}{c c} \hline \hline \((s,a)\) & The Q value of executing a in the current state s. \\ \hline \(\alpha\) & Is the learning rate, which can obtain new information and affect the existing Q value. \\ \hline \(R(s,a)\) & The immediate reward obtained by action a. \\ \hline \(\gamma\) & The key and importance of balancing current rewards with future rewards. \\ \hline \(max\) & \(Q(a^{\prime},s^{\prime})\) The maximum Q value of the new state. \\ \hline \(s^{\prime}\) & The new state after executing action a. \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Q-Learning individual meanings. Figure 2: Q-Learning. impractical in large state spaces due to excessive memory requirements for storing Q-values. DQN addresses this limitation by integrating deep neural networks, experience replay, and a fixed target network, which together enhance training stability and improve convergence efficiency. In the DQN framework [13], two neural networks are utilized: an online network, which continuously updates based on training data, and a fixed target network, which is used to estimate the Q-value formula. The target network undergoes periodic updates to reduce instability and prevent oscillations as shown in Table 2. The DQN model receives the current environmental state as input and outputs the Q-value associated with each possible action in that state. The action with the highest Q-value is selected, forming the basis of the decision-making process. A key component of DQN is the experience replay mechanism, which helps address the instability of Q-learning. Unlike standard Q-learning, which updates the neural network using only the most recent experience, DQN stores past experiences in a memory buffer and randomly samples batches for training, breaking data correlation and improving learning stability. By incorporating these enhancements, DQN is effectively applied to path determination in robotics, enabling autonomous mobile robots to learn navigation behaviors through reinforcement learning. Recent studies have demonstrated the feasibility of integrating deep neural networks into robotic navigation systems, optimizing real-time decision-making in ROS-based platforms [14]. Additionally, multi-task convolutional neural networks have been explored to enhance advanced driving assistance by embedding spatial coordinate data for robust inference [15]. Before deployment, the AMR undergoes extensive training in a simulated environment, where it explores its surroundings and learns optimal actions based on a reward function. The performance of the trained model is evaluated based on the robot's ability to navigate toward a target location while avoiding obstacles. This approach allows the AMR to autonomously select the most efficient actions based on a DQN-based scoring system, enhancing its ability to avoid obstacles and precisely reach charging stations [16]. Although alternative reinforcement learning (RL) approaches, such as Proximal Policy Optimization (PPO), Soft Actor-Critic (SAC), and Deep Deterministic Policy Gradient (DDPG), have been widely applied in robotic control, DQN was selected for this study due to several key advantages. First, DQN is inherently designed for discrete action spaces, making it particularly suitable for AMR docking and path planning tasks, where the robot selects from a predefined set of movement actions. In contrast, PPO, SAC, and DDPG are primarily designed for continuous control problems, requiring additional modifications, \begin{table}
\begin{tabular}{c c} \hline \hline s & Number of states \\ \hline \(\alpha\) & Number of actions \\ \hline Q-Learning & Deep learning \\ \hline Q-Value & Deep learning results \\ \hline \hline \end{tabular}
\end{table} 
Table 2: DQN individual meanings.