## 3 Macro-Control and Micro-Autonomy Pathfinding Strategy
### Distributed Planning Strategy
#### 3.3.1. PER-D3 QN Reinforcement Learning  
The Prioritized Experience Replay Double Dueling Deep Q-Network (PER-D3 QN) is an improved version of the Deep Q-Network (DQN) algorithm [42]. This method integrates the advantages of Double DQN [43], Dueling DQN [44], and Prioritized Experience Replay (PER) to enhance convergence speed and improve training stability.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Situation** & **The Angle Between AGV’s** & \(p(n)\) **Direction and Its Target** & **(Penalty)** \\ \hline The AGV’s target is directly in front of AGV. & angle = 0\({}^{\circ}\) & 0 \\ The AGV’s target is directly behind AGV. & angle = 180\({}^{\circ}\) & 1 \\ The AGV’s target is directly on the left or right side of AGV. & angle = 90\({}^{\circ}\) & 1 \\ The AGV’s target is on the front-left or front-right of AGV. & 0\({}^{\circ}\) \(<\) angle \(<\) 90\({}^{\circ}\) & 1 \\ The AGV’s target is on the rear-left or rear-right of AGV. & 90\({}^{\circ}\) \(<\) angle \(<\) 180\({}^{\circ}\) & 2 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Directional Penalty.  
By separating state value and action advantage, PER-D3 QN efficiently processes states where certain actions are irrelevant or redundant, reducing unnecessary computation and improving learning efficiency, and increasing training stability.  
However, directly summing the state value and advantage value may introduce bias. To mitigate this, Dueling DQN applies the following correction formula:  
\[Q(s,a)=V(s)+(A(s,a)-\frac{1}{|A|}\sum_{a^{\prime}}A(s,a^{\prime})) \tag{12}\]  
where \(|A|\) represents the number of actions; \(V(s)\) represents the state value in state \(s\); and \(A(s,a)\) represents the advantage value of choosing the action \(a\) in state \(s\).  
To address the issue of overestimation in DQN, PER-D3 QN decouples action evaluation and selection. The Q-value is still performed by the target network, but action selection is handled by the evaluation network.  
The target Q-value is updated as follows:  
\[y_{i}=\begin{cases}r_{i}&\text{if }done=true\\ r_{i}+\gamma\times Q(s^{\prime}_{i^{\prime}}argmax_{a^{\prime}}Q(s^{\prime}_ {i^{\prime}}a^{\prime};\theta);\theta^{\prime})&\text{if }done=false\end{cases} \tag{13}\]  
where \(r_{i}\) is the reward; \(\gamma\) is the discount factor; \(\theta\) represents the parameters of the evaluation network; \(\theta^{\prime}\) represents the parameters of the target network; and \(done\) is the termination flag of decision chain. If \(done\) is true, it means that the AGV has reached the target.  
The loss function \(loss(\theta)\) is defined as the mean squared error between \(y_{i}\) and the Q-value of the evaluation network, as follows:  
\[loss(\theta)=\frac{1}{M}\sum_{i=1}^{M}(y_{i}-Q(s_{i},a_{i};\theta))^{2} \tag{14}\]  
where \(\theta\) is updated with the aim of minimizing the loss function. However, \(\theta^{\prime}\) is not updated immediately. Instead, every \(N\) step, \(\theta\) can be copied to the target network, which enhances the training stability.  
In standard DQN, Experience Replay samples experiences uniformly at random. PER assigns priorities to experiences so that samples with greater learning potential are used more frequently. The probability that an experience is sampled is given as follows:  
\[P(i)=\frac{p_{i}^{\alpha}}{\sum_{k}p_{k}^{\alpha}} \tag{15}\]  
where \(\alpha\) controls how much prioritization is applied (if \(\alpha\) = 0, the sampling is uniform), and \(p_{i}\) is the priority of an experience. The priority of each experience is determined using the Temporal Difference (TD) error, which measures the discrepancy between predicted and actual Q-values, as follows:  
\[p(i)=|y_{i}-Q(s_{i},a_{i};\theta)|+\eta \tag{16}\]  
where \(\eta\) is a small value to ensure that all experiences have the opportunity to be sampled.