## 4 Result and Discussion
### PER-D3 QN-EDAgger Model Training  
To validate the effectiveness of different learning components, we conducted an ablation study comparing the following:  
1. Full PER-D3 QN-EDAgger (complete model);
2. PER-D3 QN-EDAgger without A* rewards (removing the incentive for following the A* pre-planned path);
3. PER-D3 QN without IL (removing the DAgger-based expert demonstrations; reinforcement learning alone);
4. PER-D3 QN with pre-training using DAgger (expert demonstrations only used for initialization). The training map environment is based on Figure 2 and generates different scenarios by varying the number of AGVs \(n_{agv}\) and the shelf density coefficient \(n_{shelf}\). The initialization of the experimental environment is conducted through the following steps: 1. Shelf Initialization: In the beginning, the shelf nodes generate shelves based on random numbers in \([0,1]\) to reconstruct obstacle layouts. If the random number is less than the shelf density coefficient \(n_{shelf}\), the cell is marked as occupied by a shelf; otherwise it remains free, allowing the AGVs to pass freely. During training, \(n_{shelf}\) is sampled uniformly from 0.1 to 0.8. Only natural, the larger \(n_{shelf}\) is, the higher the obstacle density of the environment. 2. AGV Placement: The number of AGVs \(n_{agv}\) is chosen uniformly from 1 to 20. Except when \(n_{agv}=0\), one AGV serves as a dynamic obstacle, making unpredictable random moves. In the environment configuration, all AGVs are initially placed at random in free-road cells. 3. Task Generation: Each training episode randomly creates between 10 and 30 transport tasks, each with exactly two key waypoints. For each task, one occupied shelf cell and one free shelf cell are selected without replacement as the start and end points. If there are too few of either shelf cell, the environment well be reinitialized. Because free-road cells are reserved in advance, no task is ever rendered unreachable. Every 10,000 training steps, a test run is executed, and the average reward is recorded. During testing, both the number of tasks and the number of AGVs are fixed at 10, while all other parameters match the training configuration, the result is shown in Figure 6.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Value** \\ \hline Discount Factor \(\gamma\) & 0.99 \\ Greedy Factor \(\epsilon\) & 0.9 to 0.1 (over 50,000 steps) \\ Prioritization Factor \(\alpha\) & 1 \\ Factor \(\eta\) & 0.1 \\ Learning Rate & 0.0003 to 0.00001 (over 50,000 steps) \\ Batch Size & 128 \\ Experience Replay Buffer Size & 100,000 \\ Target Network Update Frequency & Every 2500 steps \\ Train Total Steps & 1,000,000 \\ Warmup Size & 10,000 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Hyperparameters.  
From the results, PER-D3 QN without IL performs poorly in both convergence and stability, while the imitation learning component significantly accelerates model training. Convergence refers to learning efficiency, and stability refers to the robustness to environments.  
PER-D3 QN-Dagger demonstrates the best convergence performance but lacks stability compared to PER-D3 QN-EDAgger. This is because PER-D3 QN-Dagger overfits to A* expert demonstrations, leading to insufficient dynamic obstacle avoidance. This highlights the ability of the \(\epsilon\)-Dagger strategy to adapt and surpass the flawed expert strategy.  
The full PER-D3 QN-EDAgger shows slightly a better convergence than its variant without A* rewards, but their stability is comparable. This suggests that additional A* rewards help mitigate the sparse reward issue in the early stages of training, but have limited impact on model performance in the later stages.