## 6 Results and Discussion  
### Reward Design and Environment Setting  
The details of the rewards for the high- and low-level learners are designed as shown in Table 3. The other parameters are shown in Table 4. The agent should finish all tasks within the given \(budget=100\) time steps. The learning parameters in Q-learning are set as \(\alpha=0.5\), \(\gamma=0.9\), and \(\epsilon=0.2\). Figure 7 shows how \(\alpha\), \(\gamma\), and \(\epsilon\) affect the learning process. As shown in Figure 7a, the learning rate \(\alpha\) influences the convergence speed but does not affect the final convergence reward. In this paper, we set \(\alpha=0.5\) as a tradeoff between learning stability and speed. The discount factor \(\gamma\) has a significant impact on the final convergence reward, as shown in Figure 7b; here, we set \(\gamma=0.9\), as it achieves the highest reward. The exploration rate \(\epsilon\) in the \(\epsilon\)-greedy policy is typically set to 0.2; Figure 7c confirms that this value provides a favorable balance between exploration and exploitation. The risk threshold \(\theta=0.1\).  
We verify the proposed FSA-RL algorithm in dynamic environments, as shown by the different configurations in Figure 8. In Figure 8a, three workstations, six tasks, and two forklifts are considered. Six doors are set in the environment. To increase the environmental uncertainties, we then increase the number of tasks to twelve, with six workstations and four forklifts. To model the environment uncertainties, 2000 simulations are performed and the risk distribution \(R_{isk}\) is calculated. During the simulation, \(p_{f}=1.0\) and \(p_{w}\approx 0.0\) for the motion model of the forklifts. Note that the uncertainties are caused by a forklift moving one step or two steps ahead, which we randomly simulate. The start time of the agent is set as \(t_{0}=0\), while the forklifts start moving from time step \(t=30\).  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Parameter** & **Value** & \\ \hline \(budget\) & 100 & Section 4 \\ \hline \(p_{f}\) & \
\begin{tabular}{c} 1.0 (randomly move one \\ step or two steps) \\ \(p_{w}\) \\ \end{tabular} &
\begin{tabular}{c} 1.0 (randomly move one \\ step or two steps) \\ \(\approx\)0.0 \\ \(\approx\)0.0 \\ \end{tabular} & Section 5.1 \\ \(\text{simulation times}\) & 2000 & \\ \hline \(\alpha\) & 0.5 & \\ \(\gamma\) & 0.9 & \\ \(\epsilon\) & 0.2 & \\ \(\text{cimputation\_budget}\) & 1000 & \\ \hline \(\theta\) & 0.1 & low-level \\ \hline \hline \end{tabular}
\end{table}
Table 4: Parameter settings.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Types of Reward** & **Value** & **Learner** \\ \hline \(E\) & 1.0 & high-level \\ \(rE\) & 0.0 & high-level \\ \(step\_cost\) & \(-0.1\) & high-/low-level \\ \(goal\_reward\) & 1.0 & low-level \\ \(Penalty\) & \(-1.0\) & low-level \\ \(c\_cost\) & \(-1.0\) & low-level \\ \hline \hline \end{tabular}
\end{table}
Table 3: Reward design in the high- and low-level learners.  
We conduct 100 simulations and evaluate the proposals by the average conflict number, average task success rate, average reward, and average length of generated paths. Moreover, we compare the results with the RL method.  
Figure 7: Parameter investigation, showing the configuration with one task and two forklifts: (**a**) learning rate \(\alpha\), (**b**) discount factor \(\gamma\), and (**c**) threshold \(\epsilon\) in \(\epsilon\)-greedy policy.