## 5 Proposals
### Low-Level: MDP Learner  
The lower level consists of a local MDP learner for learning an optimal control policy and generating an optimal local path. We model the environment as \(\mathcal{M}_{l}\), defined by \(\mathcal{M}_{l}=<S_{l},A_{l},P_{l},R_{l},s_{l_{0}},s_{l_{g}},R_{isk}>\), where:  
* \(S_{l}=S\) is the state set
* \(A_{l}=\{forward,backward,right,left,wait\}\) is the action set
* \(P_{l}:S_{l}\times A_{l}\times S_{l}\rightarrow[0,1]\) is the state transition probability
* \(R_{l}\) is the reward function with \(R_{l}(s_{l},a_{l},s^{{}^{\prime}}_{l})\rightarrow\mathbb{R}\), where \(s_{l},s^{{}^{\prime}}_{l}\in S_{l}\) and \(a_{l}\in A_{l}\)
* \(s_{l_{0}}\) is the initial state and \(s_{l_{g}}\) is the goal state, both of which are provided by the higher level
* \(R_{isk}\) is the risk set of the environment. The environment of the lower level is a 2D grid world. The state set \(S_{l}\) contains all vertices of the environment. Starting from the given initial state \(s_{l_{0}}\), the agent chooses an action \(a_{l}\in A_{l}\) and explores the environment to gain experience in the form of a \(Q\)-table. The reward function \(R_{l}\) is designed as follows:  
\[R_{l}(s_{l},a,s^{{}^{\prime}}_{l})= \begin{cases}goal\_reward,&\text{if }s^{{}^{ \prime}}_{l}=s_{l_{g}},\\ step\_cost,&\text{otherwise.}\end{cases}  \tag{14}\]  
To enhance the efficiency of learning with uncertainties, we modify the reward function \(R_{l}\) (Equation (14)) as follows:  
\[R_{l}(s_{l},a,s^{{}^{\prime}}_{l})= \begin{cases}goal\_reward,&\text{if }s^{{}^{ \prime}}_{l}=s_{l_{g}},\\ step\_cost+u(s^{{}^{\prime}}_{l},t^{{}^{\prime}}),&\text{otherwise,}\end{cases}  \tag{15}\]  
Figure 5: State transitions. where \(t^{{}^{\prime}}\) is the time step at which the agent transits to \(s_{l}^{{}^{\prime}}\) and \(u(\cdot)\) calculates the reward from environmental uncertainties. When an agent approaches a door, it sends a request and the door control system may return an a message \"Y\" to open the door or \"N\" to refuse. However, whether or not a door will be available in the future is unknown. Thus, we consider the future uncertainties \(R_{isk}\) to predict whether or not a door could open. For each door \(n\),  
\[d_{n}^{t}= \begin{cases}1,&\text{if }r(B_{n},t)<\theta,\\ 0,&\text{otherwise},\end{cases}  \tag{16}\]  
where \(\theta\) is a risk threshold. As shown in Equation (16), when the risk of door \(n\) is less than \(\theta\), we consider that the door could be passed though with a higher probability. When \(r(B_{n},t)\geq\theta\), this means that passing through the door will pose a significant collision risk which is unacceptable; in this case, the door refuses to open, and we define \(u(\cdot)\) as  
\[u(s_{l},t)= \begin{cases}Penalty,&\text{if }d_{n}^{t}=0,\\ r(B_{n},t)*c\_cost,&\text{otherwise},\end{cases}  \tag{17}\]  
where \(Penalty\) is a negative constant value and \(c\_cost\) is the collision penalty. Here, \(u(s_{l},t)\) captures the influence of environmental uncertainties. Specifically, the robot receives a penalty if it is refused passage through a door when it is indicated that proceeding may pose a collision risk with a forklift. It is important to note that a collision may still occur even if the door system grants access (i.e., opens the door), as the door-opening decision is based on a threshold condition \(r(B_{n},t)<\theta\). To reflect this residual risk, the robot receives a reward proportional to the predicted risk value, provided by \(r(B_{n},t)\cdot c_{c}ost\). This design enables the local learner to utilize the predicted risk distribution \(R_{isk}\) to learn a safer path. Both the high- and low-level learners adopt a Q-learning mechanism to learn the optimal policy. The learning structures of these learners are shown in Algorithms 1 and 2, respectively. Figure 6 illustrates the interaction between the high-level and low-level learners in learning the optimal global path for executing multiple tasks.